For every PM metrics are crucial, normally these are about things like user success or value creation. For Inverted PMs metrics are tricky. But there is one metric…. which rules them all: Damage. In this post we go into why damage is such a good metric and how to use it.
I am an Inverted Product Manager (PM). Inverted PMs work on problems like security, abuse, infrastructure where our success is when nothing happens. Our strategy is different, the way we communicate is different, our users are different and our metrics are different. All of this has consequences for how we work. Read the original post if you haven’t already. In this post, I go deeper into how we can solve some of the problems we have with metrics and introduce a critical concept for inverted PMs: The Damage Metric.
Problems with metrics for inverted PMs
Traditional PMs would look at metrics to help them develop their product: DAU, Revenue, clicks, conversions etc. But as we discussed in previous posts, for Inverted PMs, everything is different like our metrics.
We as inverted PMs try to reduce the badness in our products: bad content, fake purchases, account takeover etc. We do this (generally) with detection and prevention efforts trying to reduce the amount of badness. But, prevention and detection efforts are problematic for our metric. These move the actual number of badness detected in opposite directions. Prevention tries to lower the amount of observed badness. Efforts for detection tries to increase the observed badness we find. So efforts in these areas try to pull the same line in opposite directions. Only if the people working on detection and prevention efforts work well together can this work, otherwise it could lead to tensions between them like always when 2 teams try to pull something in opposite directions. So would there be something that we could find that would move in the same direction when affected by both prevention and detection efforts, enter: the Damage metric.
The damage metric
At the end of the badness funnel, AKA the Critical Attacker journey (or CAJ), the bad actor does the badness: post bad content, malicious review, fake purchase, download someone’s pictures etc etc. Any action the bad actor does can be considered damage. Normally one critical attacker journey (CAJ) leads to several damaging actions. We can start by counting each action as 1. I will discuss limitations later.
Examples:
- An abuser goes to a social media platform, posts 2 fake comments, uploads 1 scammy video and does like-boosting on 3 videos. We could count this as 6 damage done in total.
- An attacker breaks into an account on an eCommerce platform, they change the credit card on the account to a stolen one and then do 3 purchases. We could count this as 6 damage done in total.
These are just examples, you can think about the critical attacker journeys (CAJs) in your product and define this for your own product. But now that you understand how it works, let’s unpack the pros and (of course) cons of this metric.
Advantages of the damage metric
The damage metric moves in the same direction regardless of prevention or detection efforts.
Prevention lowers damage
If prevention efforts prevent a CAJ, it also prevents the damage this attacker would do.
We would see a lowering in overall amount of damage detected across all users and products. → Lower observed damage
Detection lowers damage
If a detection effort finds an attacker (assuming in real time), it effectively disrupts the CAJ while the bad actor is trying to do the damage. This means that any damage beyond the point of detection that this attacker would have done is also prevented from happening, again across all users and products. → Lower observed damage.
All efforts lower damage
Damage solves one of the inverted PM metric problems, it being pulled in opposite directions by different efforts. Damage metric moves consistently in the same direction.
Allows identifying measures that help victims
Focusing on damage as a metric forces you as a product manager to figure out what truly are the bad things that are being done to your users.
Rather than looking at and tagging factors or abuse patterns and trying to find what’s happening most often, you’re actually challenged to think about what’s most harmful for your victims, your ecosystem or your infra. It might also highlight places that you are not yet tracking where you are not yet finding abuse or malicious activity.
In the end, tracking damage can even help you to identify features that automatically undo damage, such as cleaning up bad images or helping merchants deal with fake purchases. You might even be able to create features that help users who are victims of account takeovers to undo some of the actions that the attackers did to their accounts. These features undo damage that was done before (at least partially).
Tracking damage helps you focus on what the most impactful launches can be in terms of helping your victims. You can prioritize whatever feature reduces the most damage.
Limitations of the damage metric
Of course the damage metric is not perfect and comes with its limitations.
Subjectivity of Damage
Damage can be subjective, meaning how bad harmful actions are, can vary. Not all instances of a damaging action are created equal. For example, posting a harmful image might be considered much worse than posting a mildly offensive comment. Similarly, a malicious purchase of $100 could be significantly more damaging than a fake purchase of $1. The impact of damage can depend on factors such as the target audience, the potential consequences, and the context in which the action occurs.
Inconsistency of Damage
Another challenge with the damage metric is that not all damaging actions are comparable. It’s difficult to directly compare actions that involve different types of harm or target different groups of users. For example, is posting a bad image as bad as stealing a user’s personal information? This inconsistency makes it challenging to create a unified metric that accurately reflects the severity of all types of damage.
Fluidity of Damage
The bad things that attackers can do is constantly evolving, meaning that new types of damaging actions will emerge over time. This can make it difficult to keep the damage metric up-to-date and ensure that it accurately captures the full range of badness your users, ecosystem or infra is exposed to. As new threats arise, inverted PMs will have to update the damage metric or adjust the weighting of existing categories. I suggest you create a version of the metric and update the metric once or twice a year.
4 steps to implement the Damage metric
To implement the damage metric, use the following steps:
1. Identify the damage
First find all the potential harmful actions that can occur within the critical attacker journeys you’re responsible for. These actions could include service disruptions, harmful content uploads, fake purchases, account takeovers, or any other malicious activities relevant to your specific product.
2. Start measuring
Second, now that you’ve identified these damaging actions, you can write code to start measuring their occurrence. Make sure you capture damage correctly. If you count damage in the wrong part of your stack you might miss other routes to the same back-end and miss some volume of damaging actions. The reverse can happen also. Data quality and continuous monitoring of your implementation are important.
3. Decide your assessment framework
Third, decide how you want to assess damage. You might consider a simple raw count, categorizing actions, applying weights or converting all damage into a dollar value for comparison. Create your own framework that works for you and helps you with the 4th step.
4. Create a Damage-based product roadmap
Fourth, use the results of your damage metric to identify and prioritize prevention, detection and other measures that can drive it down. either by eliminating entire CAJs or disrupting them at specific points. Using damage allows you to build a more objective product roadmap.
5. Update the metric
Periodically review the damage metric. Add new actions that you learn off from out of bounds research or from your community. Updated your assessment framework to incorporate new knowledge. Remember the main goal is to make good product trade-off decisions, not to be perfect.
Damage is a unique Inverted PM tool
Inverted PMs are unique and so require unique concepts and solutions. The Damage Metric is one of these unique concepts that can be really powerful for Inverted PMs. By focusing on the actual bad things that might be happening, you can gain deeper insight into what the right things to build are to defend your users or ecosystem. By implementing and constantly updating the damage metric, we as Inverted PMs can really give the attackers we face a bad day, and at the same time prevent a lot of badness from happening to those we protect.
Because remember… saving one is worth it !
