Revenge of Research Methods

👋 Hi, welcome to Signal Lab where the team at Innate Language Processing digs deep into language & behavior, the limits of technologies, and the connection between what people say and do. For venture investors and startups.

In deal sourcing and screening, founder trait optimization now plays an expanding role, and one that is growing in sophistication. Take for instance Notion’s Founder DNA (Grit, Discipline, Self-belief, Execution, Leadership, and Vision and Purpose) or EQT’s Founder Six (Lightning Focus, Resilience Alchemy, Magnetic Leadership, Fearless Drive, Progressive Explorer, Transparent Self-Awareness), both drawn over three years of research,¹ and developed by the same company. (You can see the common threads, even if there are individual differences: e.g., “self-belief” is not the same as “transparent self-awareness”; “discipline” is not the same as “lightning focus”.)

The most common objection raised to otherwise impressive data-driven founder criteria is the validity of the data source itself. As they say, garbage in, garbage out.

These psychology-based founder criteria – currently used to screen in/out founders for multi-stage venture firms -- are most commonly propagated on self-reports. Founders are given a test (you can take one if you have a spare 15-20 minutes), and enjoined to respond with honesty.

Academics who constantly problematize and flag bias are, we can all agree, annoying. But in this case, the flag must be raised: self-reports are a form of data notoriously subject to bias (e.g., self-serving bias, recall bias, reference bias, social desirability bias, etc. etc.).

And bias is the biggest fly in the ointment of judgment and decision-making. It’s an established fact, in venture specifically.

One study examining 16,000 early stage investments found that most bad investments were attributable to over-indexing on founder attributes (particularly charisma), and another study found this same problem also influenced investors to pass on good investments – no charisma, no investment. This is not to say founder assessments are inherently misleading but rather that they are particularly vulnerable to cognitive bias.

Less bias, more alpha. For example, a study recorded experienced investors who showed negligible signs of cognitive bias at an IRR of 22.75%, while an algorithm achieved IRR of 7.26%, and a set of 255 angel investors averaged 2.56%. Experience was important, but bias reduction more so: experienced investors with clear signs of bias performed at 2.87% IRR.

So, the highest returns (far and away) can be delivered by experienced investors with negligible bias. And a data-driven approach is usually understood to reduce human bias, so far so good. But when bias is in the data collection, the benefits are minimized. Allowing bias in, particularly at the top of the funnel, is inviting that drop, from 22% to 2%.

And of course, self-reports are easily gamed, as is widely known throughout psychology, but justified because of their ease and accessibility as a data gathering approach. On the front page of most of these founder tests are explicit statements discouraging dishonesty. “The most common mistake people make with psychometric assessments is trying to answer strategically. Do not do that here.” Pinky promise!

The instructions state that, “gaming the assessment defeats the entire purpose”. For the investor, yes. But if it gets the founder funded, not really.

The results are predicated on an assumption of honesty, an assumption that — given what’s on the line and the prevalence of these criteria online — isn’t safe to make. Founders, in their hope of success, may believe they embody successful traits when they do not. Or they may simply attest it, knowing full well they do not. It raises the question, though it is taboo: are founders allowed to lie?

And sorry, but how easy is this to game:

Meanwhile, a spate of other services has popped up to meet the demand for founder assessment, also (always) based on the founder’s self-reports.

If you agree with NfX that “a Founder’s mental approach is the greatest point of leverage in a company’s success or failure. We’ve seen time and again: the way you think determines how you do”, then you will ignore these methodological quibbles, take the self-report results and run with them. It’s irresistible. But is a self-report the best way to access the founder’s ‘mental approach’?

Ironically, at the top on NfX’s list of twelve mental models from which the above quote was drawn is: “1. Start with Language. Language is at the core of your company, even before product….Our reality is structured by language…. Language is the center of your company, it’s not something you add later.” So why not use that language in diligence??

In short, self-reports are not the best dataset for these advanced analyses. They multiply bias and risk accuracy. This is a point in favor of behavioral linguistic intelligence, no self-reports, just pre-existing language samples – artifacts representing significant subconscious structural and grammatical choices that are even more revealing and reliable.

Next time, we’ll talk about just how deep cognitive bias goes, where it turns up, and how you can root it out.

See you next month.