Data-driven off a cliff - Anti-patterns in evidence-based decision making

Event Schedule:
19:00 Door opens & food and mingling - Please eat while it's fresh!
19:30 Tech Talk (1hr)
20:30 Q&A and further mingling and beer
21:00 Door closes Event Schedule

Metrics and analytics are crucial for building successful products at scale. They're a powerful weapon. You can hit your target, or you can blow off your own foot. You can accidentally or intentionally misuse them in all sorts of ways, hence the cliche about lies, damned lies, and statistics.

In this talk, I'll explore some of the ways you can use data to drive you off the cliff. I'll talk about anti-patterns in evidence-based decision-making. Sometimes it's a lack of math or statistics. Sometimes not understanding your domain or how a data-driven approach should be applied. Sometimes it's not appreciating the limits of what data. And sometimes it's that empiricism is unnatural, inhuman, and inhumane. You can build an exquisitely bad decision on a rock-solid foundation of data, and I'll teach you how.

In spite of that, data-driven decision making is not quite completely awful. Fallacies, biases, misconceptions, and over-confidence are the pernicious diseases that lead to many bad decisions, but you can cure them, or at least treat them. It's still possible to make good decisions, like your excellent decision to come to this talk. In spite of all the flaws, there is still good inside the data.

I'll talk about why and how Indeed measures despite the pitfalls, and how data-driven experimentation has been a critical factor in our dramatic growth. I'll talk about some of the missteps we've made in using data and what we learned from them. I'll abuse concepts from physics, pomology, and quantum hermeneutics to help you reason about metrics and achieve better results. You'll leave this talk smarter than you came in, and with evidence to prove it*.

Ketan Gangatirkar is a software engineering manager at Indeed. For the last 7 years, he's been helping people get jobs. He used to think he had great ideas, but being wrong all the time cured him of that. Now all he wants is evidence. He was analytical before it was cool. It's not cool, and it never has been, but he's still hoping. Someday. Someday