ARTICLE: Metrics–an IMRA comparison

My readers will have to forgive me for having to go a bit technical on you today, but I want to get you thinking about how you best measure your performance in general and what this means for measurement methods such as “% of Win Time” and “Enduro Points” on the IMRA website.

I am from an IT and science background and so I hear certain terms bandied about on a daily basis and these have become just another word in my vocabulary whereas they may not mean a lot to people from different works of life. One of these words is metric which Wikipedia defines such: “A metric is a measure for quantitatively assessing, controlling or selecting a person, process, event, or institution, along with the procedures to carry out measurements and the procedures for the interpretation of the assessment in the light of previous or comparable assessments.”

“WTF!” May be your natural response to seeing this definition, more colloquially a metric is a single measurement of performance (such as running speed) and multiple “metrics” are actually multiple separate measurements (such as “maximal pace”, “average pace”, and “average heart rate”).

Measurements and metrics for dummies

A measure is the key word here and in science in general: E.g. you have to measure a single variable alone in order to be able to isolate it and understand what influences it. When I say variable I simply mean thing such as “pace”, “body weight”, “temperature” and so on. Each of these factors can influence your running performance on a given day and each of them can be measured separately.

One or multiple variable metrics

Sometimes, multiple metrics are conflated into one such as the (in)famous cycling doctor Michele Ferrari’s “VAM” (velocity Ascended, Metres per hour). To get to this “metric”, you need to measure several things: metres ascended, minutes it took to ascend, and the gradient of the incline. All of these three are separate variables and if they change, the metric changes. However, all of these are objective, that means if you have precise measurement equipment you will have an accurate representation of exactly what occurred (e.g. you really did ascend 200m in 6 minutes up a 10% gradient).

As you will undoubtedly note, the drawback of Ferrari’s metric is that it takes longer to calculate it and it is difficult to understand what “VAM” is if you don’t read up on it. On the other hand “minutes ascending” is straightforward to anyone. So as a general principle, I prefer metrics who measure only one variable and luckily most running metrics are of this character including favourites such as position, time, pace, distance run, total ascent and so on.

Objective measures versus subjective measures

The second key principle is objectivity versus subjectivity. An objective measure is more precise than a subjective because it describes an observed unbiased value such as “pace”. If your Garmin is working correct and it shows you ran at an average pace of “4:32min/km”, then that is how fast you ran. End of discussion.

A subjective measure on the other hand is trying to quantify something “immeasurable” and as such is only as precise as the system used and the person using it. It is not free from bias and not as reliable as objective measures. It can never provide certain evidence of anything but can provide clues to the truth. Good examples in running is Borg’s “Rate of Perceived Effort” where you get a table that describes how hard a given intensity feels and you then rate it from 1-10 (in some cases the scale is different). The problem is that everyone has their own interpretation of the descriptions and may be biased in their choice as well (if I am told I should reach my maximal exertion level in a test and I am told that is 9 to 10, I will likely rate the exertion 9 or 10 when I quit or otherwise admit to myself that I did not give my maximum).

The only way to ever measure factors such as exertion objectively is if we could monitor the pain response of the brain somehow (one day we may). I hope this show the difference between subjective and objective and why you should favour objective metrics over subjective metrics when trying to evaluate your training.

Simplicity and comprehension

Another important thing for a metric is that it is understandable, usually because it is a simple variable to understand. If I tell you that I finished 5th, you understand exactly what that means although you may feel like inquiring “out of how many?” and “how good was the field”. To the first question you have another objective answer (e.g. “oh, 120 turned out”) whereas the second answer is subjective and biased depending on how it is answered (if you answered it with a reply such as “50% of the field finished within 5 minutes of the course record”, you are actually objectively assessing the strength of the field. Few people will answer the question in those terms!).

Let’s look at how the new Enduro system (in principles, I won’t go into the specific mathematics here) and the older “% of win time” in terms of the three criteria: “number of variables”, objectivity/subjectivity and simplicity.

“% of Win Time”, a winner?

A quick look at “% of Win Time” reveals a few things: It is objective (e.g. it basically calculates how far behind the winner of any given race you finished). We cannot tell why you finished a certain percentage of time behind the winner’s time, but the “% of winning time” states a clear unambiguous fact about the race on the day.

Secondly, my anecdotal conversations with people suggest that “% of winning time” is easier to understand. When I first logged on to the IMRA website five years ago, I instantly understood what “% of win time” meant with no further explanation required. It was no more a difficult concept to understand than the other objective metrics on the website (your position in the race and your finishing time).

Finally, “% of winning time” is only measuring one thing: The difference between you and the first runner. While you need some calculations to arrive at the percentile format, you are still looking at a relatively straightforward concept.

So far so good for “% of Win Time”.

Enduro Points

Enduro points represent a very different type of metric than “% of Win Time” in that it attempts to provide a fairer representation of performance by assigning a race a number of “base points” based on the perceived difficulty of that race and then scoring the runners performance as combination of their performance on the day and the toughness of the course (in short: “performing at the same level in a tough race compared to a race perceived to be inferior will yield more points”).

Some of the factors used to determine points are purely subjective (“terrain” because we cannot quantify terrain difficulty by numerical values at current) and others are based on objective measures (elevation and distance). As some people debating this on Boards have pointed out, the decision on whether a longer race is harder than a shorter race is subjective and you would expect different answers if asking Sebastian Coe rather than Scott Jurek (or not, we cannot know for sure). Thus to accept the Enduro point system as an accurate reflection of performance, you need to agree to all underlying assumptions.

As mentioned, subjective measurements such as this are useful because you would normally use them to go on and test the assumptions you have put forward objectively afterwards. There is obviously a lot of work ahead to test this as we currently have no satisfying way of measuring “hardness” or “exertion” induced by a race, so you are left relying on an assumption that cannot currently be satisfyingly tested.

Enduro points are also a metric comprised of multiple variables (the afore-mentioned distance, terrain and elevation) which are inserted into a mathematical formula to generate your output. This means to fully evaluate the metric you need to understand (and agree) with the calculation employed and to use it in performance evaluation you would need to consider whether the factors used in the calculation are relevant to you (this criticism also goes for “% of win time” because you may have no interest in the winner’s time but only in your own time but you only have one variable to take out in that case).

Questions ahead – tomorrow…

So where does this leave us? Which metric is “better” (and does it make sense to talk of “better” in this context). What is the purpose of these metrics in the first place, who uses them, what do they show and what audience benefit the most from them? And finally is there a “third way”?

I’ll turn to these questions tomorrow as I am running out of time and space!

Comments