Member-only story
Platforms Are Still Flying Blind When It Comes to Societal Impact — Better Metrics Could Help
Metrics are key to how product teams at tech companies function
After the 2020 election, a Twitter dashboard that I first prototyped four years before started going wild. It estimates misinformation prevalence by monitoring “the percent of retweets and likes pointing toward domains that had made a habit of sharing misinformation.” This metric had been going up throughout the election cycle from a low around 10% up to almost 20% on November 3rd. And then it jumped wildly to 30% over the next week and stayed there for almost a month. Something was likely very wrong.
Tracking this sort of change is a valuable step toward understanding the platform’s impact. It was almost the simplest possible “health metric” — it likely should have triggered alarm bells and executive meetings — it “should” have been equivalent in importance to a significant drop in a revenue dashboard. But it was also clearly insufficiently visible.
In practice, product teams at companies like Twitter, Facebook, and YouTube are primarily rewarded for two things: moving metrics and shipping products — which then move metrics. If you want to change a system, you must understand its incentives. Since metrics are core to changing platforms, tools that make measurement or improvement easier can be one of the most effective routes to moving platforms. We can’t know if we’ve reduced the extent of misinformation or hate speech without an estimate of how much there is. Paying more attention to the type of information tracked by such metrics could also help ensure platforms can compete on characteristics more valuable to society than attention or stock value. Shortly after the 2016 election, I published a piece suggesting a focus on one such metric, to measure attention toward misinformation.
We still don’t have a great way to measure this — the metric that showed the spike of misinformation right after the 2020 election is only a very rough proxy. I developed the initial version of this independently and brought this project…
