Copilot readiness is a metadata project: why your AI gives inconsistent answers

The most common AI conversation we have does not start with AI. It starts with an organisation that enabled Copilot, got inconsistent answers, concluded the technology was not ready, and switched it off.

The technology was usually fine. What was missing was the metadata it had to reason over, and fixing that is cheaper than almost anything else on an analytics roadmap.

What Copilot is actually doing

Copilot over a semantic model is a language interface over metadata. When a user asks "what was margin by region last quarter", the system has to map those words onto fields and measures in a model, construct a query, and present a result.

Everything about the quality of that answer depends on what the model tells it:

• Are the fields named in words a human would use, or amt_2 and dim_cust_flg?
• Does each visible field have a description explaining what it means?
• Are there synonyms covering the vocabulary your business actually uses, "turnover" for revenue, "matter" for project, "spell" for admission?
• Does a measure exist for margin, or will it try to aggregate raw columns and produce a plausible, wrong number?
• Is there exactly one model that could answer the question, or five that would each answer it differently?

That last one is decisive, and no amount of prompt engineering fixes it.

The ambiguity problem

In one engagement we found five semantic models covering utilisation across a professional services firm. Each had been built for a different audience, each defined availability slightly differently, and all five were discoverable.

A natural-language question about utilisation could therefore be answered correctly by one model and incorrectly by another, with no indication to the user which they had reached. The firm's conclusion, "Copilot is unreliable", was entirely rational from where they were sitting.

Consolidating those five into one certified model delivered most of the accuracy improvement in the entire engagement. The metadata enrichment that followed was necessary, but ambiguity removal came first because enrichment across five competing models would have made all five more convincing rather than making one correct.

Measure first, then improve

Before changing anything, build an evaluation set. Ours is typically 80 to 150 real questions, collected from the people who will use the feature, each with an agreed correct answer.

Run it against the estate as it stands and record a baseline. This matters for three reasons:

1. It converts "the AI is unreliable" into a number, which can then move.
2. It reveals which kinds of question fail (aggregation, time comparison, filtering, ranking), and those categories point at different fixes.
3. It lets you set a success threshold before you start, so the go/no-go decision at the end is not a judgement call made under sunk-cost pressure.

Collect the questions from users, not from the project team. Project teams write questions the model can answer.

The remediation checklist

With a baseline in hand, the work is mostly unglamorous:

Consolidate competing models. One certified model per business domain. Retire or clearly mark the others as non-authoritative. This is political work; do it first anyway.

Rename for humans. Visible fields get business names. cust_nm becomes Customer name. Technical columns the user never needs get hidden rather than renamed.

Describe every visible field. Not restating the name, stating what it means, its grain, and any exclusion. "Net revenue, excluding intercompany and VAT, at invoice line grain" tells the language layer something Net revenue does not.

Add synonyms from real vocabulary. Sit with users and write down the words they use. Legal firms say "matter"; hospitals say "spell" and "episode"; manufacturers say "line" for both a production line and an order line, which needs disambiguating deliberately.

Complete measure coverage. Every question in the evaluation set needs a measure that answers it. Where one is missing, the language layer will aggregate a raw column and produce something plausible. This is the single largest source of confidently wrong answers.

Hide what should not be reachable. Surrogate keys, audit columns, staging artefacts. If it is visible, it is a candidate for an answer.

Mark the date table and set hierarchies properly. Time comparisons fail silently without this.

Then re-run the evaluation

Re-run the same set and publish the result to the sponsor, including the categories that are still weak. An honest "we improved aggregation questions from 48 per cent to 91 per cent, and ranking questions remain at 60 per cent" buys far more trust than a claim of general improvement.

Then publish user guidance that states what the assistant is reliable for and where to open the report instead. Counter-intuitively, documenting limitations increases adoption, because users stop having to guess whether to trust an answer.

Cost and audience

Copilot consumes capacity. Enabling it tenant-wide without measuring is how a capacity throttles unexpectedly, and the throttling affects everything on that capacity, not only the AI feature.

Enable it for a defined audience, measure the consumption, and publish the cost per active user before widening. We have never seen that cost be the blocker, but we have seen the surprise be a problem.

Where a custom assistant is justified

Copilot is strong at questions over governed analytics data, with no build cost. A custom assistant on Azure OpenAI is justified when you need something Copilot does not do:

• Retrieval over your own documents alongside the data.
• A specific workflow rather than open-ended questioning.
• Tight control of prompts, outputs and refusal behaviour.
• Integration into an application your customers or staff already use.

If Copilot covers the need, building a bespoke alternative is a straightforward waste of money, and we will say so. Where a custom assistant is right, it needs the same things any production system needs: an evaluation set, prompt and output logging, grounded citations, cost monitoring, and guardrails against prompt injection and data leakage.

The governance position

Most analytics assistants sit in the lower risk tiers of emerging AI regulation, but the transparency and documentation obligations still apply, and classification depends on use rather than technology.

What we maintain per use case: intended purpose, data sources, model and region of processing, human oversight arrangements, known limitations, and evaluation results over time. That is the documentation set a regulator, customer or risk committee is most likely to ask for, and it is considerably easier to maintain from the start than to reconstruct.

On the commercial terms: under the current Azure OpenAI and Fabric agreements, your prompts and data are not used to train the foundation models. The exact boundary, region of processing, retention for abuse monitoring, which features call which service, is something we document per deployment, so your risk function reviews the real configuration rather than a marketing summary.

What we would do first

Spend two weeks on a readiness assessment: model inventory, ambiguity analysis, metadata scoring, and an evaluation set with a baseline. In most organisations that produces a remediation backlog measured in days of work, not months, and the accuracy improvement is large enough to settle the question of whether to proceed.

What a good evaluation set looks like

The evaluation set is the artefact that turns an AI conversation from opinion into measurement, and most organisations build a bad one by accident.

Collect from users, not from the project team. Project teams write questions the model can answer. Users write "how are we doing on the Thompson account" and "why is margin down in the north".

Cover the question types separately, because they fail for different reasons and need different fixes:

Record the correct answer, not an approximation. An evaluation entry without an agreed answer is a vibe check.

Keep it stable. The value is the trend over time. Adding questions is fine; silently replacing the hard ones is how a programme fools itself.

A set of 80 to 150 questions is enough. Beyond that, the marginal question tells you less than the time it costs to maintain.

Logging, cost and the things that bite later

Three operational details that are easy to add at the start and painful to retrofit.

Prompt and output logging. You need to know what was asked and what was returned, for three reasons: diagnosing a bad answer, demonstrating oversight to a risk function, and building the next evaluation set from real usage. Log the question, the generated query where available, the answer and the user, with a retention period agreed with your DPO.

Cost attribution. Copilot consumes capacity, and that consumption competes with refresh and interactive queries. Watch it in the Capacity Metrics app during the pilot, and publish the cost per active user before widening the audience. The figure is usually modest; the surprise is what causes problems.

Refusal behaviour. Decide what should happen when the assistant cannot answer confidently. Silence is bad, a confident guess is worse, and a clear "I cannot answer that from this model, here is the report that can" is best. Where a tool offers configuration for this, set it deliberately rather than taking the default.

What this looks like as a programme

Two weeks of readiness assessment. Four to six weeks of remediation, which is mostly metadata work an internal analyst can do with direction. Two weeks of pilot with evaluation. Then a decision, taken against a threshold agreed before anyone started.

The outcome we see most often is that the remediation delivers more value than the AI feature does (better documented models, fewer duplicates, a maintained measure dictionary), and that the AI feature then works as a by-product. Clients occasionally find that mildly deflating. It is still the cheapest route to both.

It also sometimes concludes that the organisation should fix its semantic models for their own sake and revisit AI afterwards, which is a cheaper and better outcome than a pilot that fails for reasons nobody diagnosed. Book a technical review if you want that assessment done on your estate.