Most organizations choose an assessment the way they choose software. They see a demo, hear that a respected competitor uses it, and move to adopt. The tool comes first. The question it is meant to answer comes second, if it comes at all.
This is backward, and it is one of the most common patterns we see. A talent leader inherits a validated instrument from a previous role and brings it along. A vendor bundles an assessment into a larger engagement, and it quietly becomes the standard. A team standardizes on one profile because it is familiar, then applies it to selection, development, team building, and succession without pausing to ask whether a single tool can carry that much weight.
What we often see is that the instrument itself is fine. The problem is the order of operations. When you start with the tool, you inherit its assumptions instead of examining your own. You end up measuring what the instrument happens to measure, rather than what your decision actually requires.
Patrick Callahan, who leads our Assessment Services portfolio, offers a better starting point. Before you look at a single product, ask three questions. They change what you end up choosing and how much you can trust the result.
- What construct are you measuring?
- Is the instrument valid for this population?
- Who interprets the result?
None of the three is technical in a way that should intimidate a business leader. Each one is a discipline, and each one protects a decision that matters.
What construct are you measuring?
A construct is the underlying thing you are trying to observe. Not the score, and not the label, but the human quality the score is standing in for. A construct measuring personal qualities and attributes might include things like judgment under pressure, adaptability, the tendency to escalate or calm a conflict, raw cognitive ability, or the capacity to build trust across differences.
The trap is that most assessments produce clean, confident output regardless of whether they measure the construct you care about. A personality profile tells you something real about how a person prefers to work. It does not tell you whether they will make sound decisions when the stakes rise and the information is incomplete. Those are different constructs. A tool built for one will not answer for the other, no matter how polished the report looks.
So the first question forces a prior question. What are you actually trying to predict or develop? If you are selecting a VP who will lead a function through a period of rapid change, the construct that matters might be how they handle ambiguity and adapt their approach when their first plan stops working. If you are building a team that keeps colliding in the same unproductive ways, the construct is how each person behaves when conflict shows up. Name the construct first. Then (and only then) go looking for an instrument built to measure it.
Skip this step and you get a subtle failure that is hard to catch. The assessment runs, the numbers arrive, everyone treats them as meaningful, and the decision rests on a measurement of the wrong thing. The report is not wrong. It is answering a question nobody meant to ask.
Is the instrument valid for this population?
Validity is not a property an instrument carries around like a stamp. An assessment is valid for a specific purpose, with a specific group of people, predicting a specific outcome. Move it outside those conditions and its validity does not automatically travel with it.
This matters most at the top of the organization, which is exactly where the stakes are highest. Many assessments were built and normed on broad working populations. That is useful for a lot of purposes. It is a weaker foundation when you are evaluating a senior executive against the demands of a C suite role, because the sample the instrument was validated on may look almost nothing like the person in front of you or the job they are being considered for. The tool can still generate a number. Whether that number means what you think it means for this person, in this role, is a separate matter.
The population question also surfaces problems of fairness and defensibility. An instrument that predicts well for one group and poorly for another is not neutral just because it produces the same tidy output for everyone. In selection especially, where decisions get questioned and sometimes challenged, you want to know that the evidence behind a choice holds up for the actual people it was applied to.
In moments like these, the honest answer is sometimes that no single instrument is valid enough on its own. That is not a reason to abandon assessment as a practice. It is a reason to combine methods, to weight the tools appropriately, and to treat any one score as a piece of evidence rather than a verdict. The work here is not just about finding a valid instrument. It is about understanding the boundaries of what each instrument can defensibly claim.
Who interprets the result?
This is the question organizations underestimate most, and it may be the one that most separates a useful assessment from a misleading one.
An assessment result is raw material. Someone has to turn it into meaning, and that act of interpretation is where most of the value, and most of the risk, actually lives. Two people can read the same profile and reach opposite conclusions. One sees a red flag where another sees a strength that is being misread out of context. What separates them is rarely the data. It comes down to the judgment, training, and independence of the person holding it.
Interpretation goes wrong in predictable ways. A hiring manager reads a report through the lens of a candidate they have already decided to hire, and the assessment becomes decision validation rather than decision support. A well-meaning generalist overreads a single elevated score and turns a minor tendency into a disqualifying story. Someone with a stake in the outcome, consciously or not, reads the evidence toward the conclusion they need. The instrument did its job. The interpretation undidit.
This is why the question of who interprets is inseparable from the question of independence. An assessment read by someone with no stake in a particular outcome carries a different weight than the same assessment read by someone who benefits from a specific answer. An outside, trained interpreter brings expertise, and just as important, brings no reason to see what someone wants to see.
Good interpretation also connects the individual result back to the system around it. A leader does not perform in a vacuum. Their tendencies show up differently depending on the team, the pressure, the culture, and the moment. The person reading the result should be able to place it in that context, rather than handing over a number and letting the organization supply its own story.
Starting with the question
Put the three questions together and a discipline emerges. Name the construct you need to measure. Confirm the instrument is valid for the people and the purpose in front of you. Make sure the result lands with someone equipped and impartial enough to interpret it well. Only after those three are settled does the choice of a specific tool make sense.
This order may seem slower than what you are currently doing. In practice it is actually faster, because it prevents the expensive failure of running a clean process on the wrong foundation and discovering the mismatch after a promotion has been made or a hire has not worked out.
The reason so many organizations get this backward is not carelessness. It is that the market is organized around tools, and tools are easy to compare. Questions are harder. They require you to articulate what you are trying to learn before anyone can sell you the answer. That is uncomfortable work, and it is exactly the work that protects the decisions that matter most.
We think about these three questions on every engagement, because the cost of getting an assessment decision wrong is steep. It is the leader placed in a role they were never suited for, the strong candidate screened out by a misread score, the succession plan built on a measurement that never meant what everyone assumed. Assessment done well is one of the most powerful tools a leadership strategy has. Assessment chosen backward is just expensive confidence.
Want to know more about how we approach assessment? Contact us today.
