Application Kata: Comparative Estimation
Estimations are used in software development at least for assessing the value of some requirement and the budget needed to implement it. As can be seen in countless teams such estimation is very, very hard to get right, though (see for example What We Know About Duration - Part 1). Nevertheless product owners and developers are asked to provide estimates all the time.

Absolute Estimation

One popular way to estimate the presumable effort to implement some requirement is to use Story Points (SP). Instead of estimating real time in hours or days an abstract metric is used. Whether a requirement estimated at 5 SP will take 5 hours or 5 days or 10 days is not known and not important. What matters is that 5 SP is more than 3 SP and less than 10 SP.

At least that's the theory. In practice, however, whoever requires an estimation soon will start to convert SP estimates into time durations. The reasoning behind that is: “If last time 85 SP took the team 10 days to implement, then next time they will take 10 days again. Each day the team is capable of delivering 8.5 SP.”

Such reasoning is understandable – but it's utterly counterproductive. It puts pressure on development as soon as an estimate is converted into a deadline: “Oh, so you're estimating this to be 110 SP? That means you'll finish it in 13 days.”

Although SP estimation might never have been meant to be used in that way it certainly often is.

More or less the same happens if estimations are based on t-shirt sizes, e.g. S, M, L, XL.

The core problem lies in the absolute metric: A specific value (e.g. 8 or M) is used to express the size of a budget needed.

Comparative Estimation

To overcome the mapping of abstract estimations to absolute values of time and money relative estimations can be used. Instead of assigning some number to a requirement it's just compared to another requirement. The result of the comparison is an order.

Absolute estimation: requirements A, B, C are estimated at A=5, B=8, C=3.

Relative estimation: requirements A, B, C are estimated at B>A>C.

If then A+B+C takes 10 days to implement nothing can be carried over to the next estimation of X, Y, Z! And that's a feature, not a bug 😏

Relative estimation or comparative estimation works by comparing alternatives in pairs. Given requirements A, B, C, D these pairs have to be compared: A:B, A:C, A:D, B:C, B:D, C:D. That's 3+2+1=6 comparisons for 4 alternatives. More generally for n alternatives it`s (n-1)((n-1)+1)/2 = n(n-1)/2 comparisons.

For each pair it needs to be decided which alternative is deemed “larger”, “heavier”, “of more value”, “requiring more effort”… Example:

  1. A:B -> B>A
  1. A:C -> A>C
  1. A:D -> D>A
  1. B:C -> B>C*
  1. B:D -> D>B
  1. C:D -> D>C*

The resulting order: D,B,A,C

Comparisons 4. and 6. are marked with a * because a different result would cause an inconsistency. They would not even have to be made.

Since B>A (1.) and A>C (2.) B needs to also be estimated > than C (4.) due to transitivity.

Maybe then comparisons 4. and 6. should not be offered at all. On the other hand maybe it would be worthwhile to let inconsistencies occur because they flag so far hidden confusion or misunderstanding which needs to be resolved before a reasonable estimate can be delivered.

If more than one person does a relative estimation of alternatives the overall result can be calculated by counting ranks, e.g.
Rank
Estimator 1
Estimator 2
Estimator 3
1.
D
D
B
2.
B
A
D
3.
A
B
C
4.
C
C
A
The total rank of the alternatives would be the sum of their ranks, e.g. 1+1+2 for D.

A
B
C
D