How to tell if your MSP is delivering

Why this matters

Many SME relationships with an MSP are long-standing — often the result of a decision made several roles ago, never reviewed since, and embedded in how the organisation operates. The monthly invoice arrives. The tickets get closed. The board assumes it is being looked after.

Competence is rarely the issue. Most MSPs are competent at what they were engaged to do. The harder question is whether the contracted work is still the right set of work, and whether the delivery is being measured by anyone other than the party being measured.

Five questions help the board self-diagnose the relationship in an afternoon. If four or five of them can be answered cleanly, the MSP is likely delivering and the relationship can be continued with minor tuning. If exactly three can be answered cleanly, the relationship is functional with identifiable gaps worth addressing. If fewer than three can be answered cleanly, there is material value in an independent review.

1. Is the SLA the right SLA?

Most MSP contracts contain a service level agreement specifying response times, resolution times, and uptime targets. Before asking whether the SLA is being met, ask whether it measures anything the organisation actually needs.

Good test: if the SLA is entirely about ticket response and resolution times, it does not measure service quality in any business-facing way. A ticket closed in forty minutes that did not resolve the problem is a metric win for the MSP and a measurable failure for the organisation.

Better SLA elements (rarer in SME contracts, common in enterprise):

Problem-recurrence tracking (was the same issue closed three times this quarter?)
End-user satisfaction measured at closure rather than at first touch
Time to resolution for specific classes of incident, separated out instead of averaged together
Outcome measurement for enhancement requests as well as reactive tickets

Ask the MSP for the last twelve months of ticket data exported to CSV. The export alone is a useful signal — the time it takes, and any reluctance along the way, tell you something before you have read a row.

2. What does the ticket data actually say?

If the ticket-data export arrives, five patterns are worth looking for:

Recurrence clusters — the same category of issue appearing multiple times across the year. A healthy environment resolves the underlying cause once; recurrence that is never traced to a root cause is a cost the organisation keeps paying.
Severity distribution — if everything is priority 3, the prioritisation mechanism is inert; if everything is priority 1, triage is not being done at all. Either way, the classification has stopped carrying information.
Same-user concentration — if a small group of users generates a large share of tickets, ask whether that is a training need or an unaddressed technical fault. Recurring tickets from the same desk often trace back to failing hardware rather than to the user.
Time-of-closure clustering — tickets closing precisely at the SLA window suggest closures are being timed to the deadline instead of to the fix.
Root-cause coding — good MSPs code the root cause; poor MSPs code only the symptom. If every ticket root cause is “software error”, the data is not being analysed.

Reading ticket data well is a skill. Even a rough read is still useful; a partner, CFO, or IT-aware board member will see more than they expect in an afternoon.

3. Is the roadmap progressing?

Every competent MSP relationship has a roadmap — a register of improvements, migrations, and enhancements agreed between the organisation and the MSP. Usually it lives in the quarterly business review deck, is referenced verbally, and is not tracked rigorously.

The question: in the last twelve months, how many roadmap items have been closed, how many are in progress, and how many have slipped past their original date without a written re-commitment?

A reasonable picture might be roughly two-thirds of items closed on time, the rest in progress or pushed with a documented reason. A weaker one is a quarter closed, half in progress, and a quarter quietly no longer referenced.

Slippage itself is normal; projects slip for good reasons. The warning sign is the absence of a written register. If the roadmap lives only in the account manager’s head, there is no audit trail, and the organisation cannot tell the difference between a good quarter and a bad one.

4. Are the commercial terms being honoured?

Three commercial signals worth investigating annually:

Licence utilisation. Is the organisation paying for Microsoft 365 or equivalent licences it is not using? Are users assigned to tiers they do not need? Are there seats for people who have left? Exports for Microsoft 365 licence allocation, Entra sign-in activity, and group membership reveal this in a morning.
Invoice-to-contract match. Are the last three invoices reconciling against the contract? Are there line items that appeared without a corresponding change order? Are there out-of-scope hours that were not pre-approved in writing?
Vendor markup. If the MSP resells software, hardware, or third-party services, is the markup visible to the organisation? Is it within the range the organisation agreed to? Is it within the range the market considers reasonable?

The presence of commercial drift is not, on its own, evidence of wrongdoing. MSPs operate on thin margins, and licence management is genuinely hard. The governance failure is drift that nobody is measuring.

5. Is the cyber-insurance control set being substantiated?

The MSP operates most of the controls a cyber insurer asks about at renewal. The questions, and the premium, land on the insured.

Five sub-questions:

Has the MSP provided a current export of MFA coverage?
Has the MSP provided a current export of conditional-access policy state?
Has the MSP provided a dated log of a successful restore test in the last ninety days?
Has the MSP provided a current patch-compliance export?
Has the MSP provided a current EDR coverage export?

If any of the answers is “I would need to ask”, the organisation is not in a position to respond to an underwriter in the window they expect. That is the gap most worth closing before the question arrives.

What to do with the answers

If the organisation can answer four or five of the five cleanly, the MSP relationship is in good shape and a full review could plausibly be annual. A standing independent scorecard (Adeo Pulse or equivalent) gives the board light-touch confirmation without structural change.

If the organisation can answer three cleanly, the relationship is functional and there are identifiable gaps worth addressing. A Quick-Scan is designed for this situation: a targeted engagement, scoped in the written proposal.

If the organisation can answer fewer than three cleanly, an independent review is the logical next step. A Baseline Audit is the right level of engagement — focused, multi-week, wider in scope, producing a board-ready written assessment and a remediation roadmap.

In any case, the first step is the organisation asking the questions directly. The answers produced are evidence, either way.

A board-level framing

“Are we safe?” is the wrong question for the board to put to its MSP — the answer will always be yes, and it will be truthful to the limit of the MSP’s visibility. The better question is:

“Could we produce, on short notice, an independent document that explains what our MSP delivered this year and what we paid for it?”

If the answer is yes, the governance is in order. If the answer is no, the governance is not in order, and that is the specific gap an independent adviser exists to close.

What this article does not cover

The specific mechanics of dropping, changing, or retendering an MSP. That is a different conversation and a different document.
Any recommendation of specific MSPs, vendors, or products. Adeo takes no commissions and names no preferred suppliers.
Incident-response playbooks. That is a specialised field and Adeo will refer cleanly to a specialist when an incident is in flight.

Adeo is an independent, vendor-neutral IT advisory for Australian SMEs. We audit, score, and oversee what a managed service provider delivers. We do not resell, and we take no commissions. Contact: contact@adeo.au · adeo.au.