We Almost Published the Wrong Cold Email Number. Here's How We Caught It.
- Keith Mortier
- 6 days ago
- 6 min read
We built a version of this article last week comparing three cold email hook types by reply rate. Before it went out, a second read caught a problem: the number we were using for our own campaign's reply rate didn't match our own records. This is the article that replaced it, because the catch turned out to be a better example of the thing we were trying to say than the original piece was.
What happens when your own reports disagree with each other?
We found out directly. Two internal reports on the same CMMC-compliance campaign, taken a few weeks apart, covered almost identical send volume, 17,175 emails in the earlier report and 17,203 in the later one, a gap of 28 emails. The earlier report stated a 1.65 percent reply rate. The later one, a full platform export we could check line by line, showed 48 replies against 17,203 sends, which works out to roughly 0.28 percent.
That's not a rounding difference. It's nearly 6 times apart, on almost the exact same denominator, 28 emails apart out of over 17,000. At 1.65 percent, 17,175 sends should produce roughly 283 replies. The platform showed 48.
Bottom line: when two of your own numbers disagree by that much, at least one of them is wrong, and you don't get to pick the one you like better.
Which number is right, and how do you find out?
You reconstruct the denominator, or you don't use the number.
We went back to find out where the 1.65 percent came from. It traced to an earlier progress report, and we could not reconstruct exactly what it was measuring against, whether it counted replies against a different date range, folded in auto-replies, or was measured against opens rather than sends. Without that denominator, the figure isn't defensible, no matter how many other documents had already repeated it by the time we checked.
The 0.28 percent figure is different. It comes from a platform export with a reply count and a send count sitting next to each other, both auditable, both from the same source, isolated to the same CMMC campaign rather than blended with anything else. That's the number that survives the check, and it's the only one this article uses going forward.
My take: a number that's been repeated in five internal documents isn't more trustworthy than a number repeated once. Repetition isn't verification. Only the export is.
Why does this matter more for this particular article than almost any other topic?
Because this article is about measuring cold email honestly, which means it has less room than almost anything else we write to get its own measurement wrong.
We've made this point before in this same cluster:
CRM numbers are self-reported and nobody audits them, and a client's own pipeline turned out to be inflated by 40 percent once someone actually checked each record by hand. The lesson there was to audit the numbers you're being handed before you build a budget or a forecast on top of them. Publishing an unverified number about our own campaign, in an article warning readers to verify theirs, would have made us the example instead of the source.
Bottom line: if you're going to tell someone else to check their denominator, check yours first.
What's a defensible way to compare your own number to a published benchmark?
Only after you've confirmed what your own number actually measures, and even then, expect the published benchmarks to disagree with each other by a wide margin.
Instantly's 2026 benchmark report, built from billions of cold email interactions across its platform, puts the average reply rate at 3.43 percent, with a top quartile above 5.5 percent and an elite tier clearing 10 percent. Belkins' 2026 study, drawn from 7.5 million emails sent to net-new contacts with a strict replies-over-total-sends denominator, puts its 2025 average at 0.45 percent, about 7 times lower than Instantly's. Both studies are real, sourced, and correctly cited. They simply measure different populations with different methodologies, which is the same lesson from the section above showing up again at the benchmark level.
Against that range, our audited 0.28 percent sits just under the Belkins net-new average and well below Instantly's platform-wide number. We're not going to round that up. It's a below-target result on a campaign that needed better targeting or a stronger offer, and saying so plainly is the whole point of this piece.
Bottom line: don't adopt a single external number as your target before you know what it's measuring, and don't round your own number up to make a benchmark comparison flattering.
What should you actually do before you publish or act on any reply-rate number?
Three checks, in order, before the number goes anywhere near a report, a proposal, or a public article.
First, confirm the denominator. Reply rate means replies divided by emails sent, nothing else. If a document doesn't say which, treat the number as unverified until you find out.
Second, find the export, not the summary. A summary is someone's interpretation of the export, written at some point in the past, and interpretations drift the more times they get copied into a new document. The export doesn't drift.
Third, if two of your own numbers disagree, that's not a rounding issue to average away. It's a signal that one of your reporting processes is broken, and the fix is worth more than the number you were trying to publish in the first place.
My take: the discipline that catches a wrong number before it ships is the same discipline that makes a campaign worth running. Neither one is optional.
The pattern underneath all of it
Every number in this piece measures something specific, and none of them are interchangeable with the number next to it. A progress report isn't a platform export. An open rate isn't a reply rate. A platform-wide average isn't a strict net-new average. Treating any two of these as the same thing is where cold email "results" go wrong, and it happened to us before it happened to a reader.
Bottom line: we caught it before it published. That's the standard, not a bonus.
Related: B2B Lead Generation. How we structure outbound campaigns, including the reporting checks that catch a bad number before it reaches a client or a public page.
Want your own cold email numbers audited before you act on them? We'll check what your reply rate actually measures before you build a decision on top of it.
FAQ
What's the difference between open rate and reply rate in cold email?
Open rate measures how many delivered emails got opened. Reply rate measures how many sent emails got a response. They use different denominators and answer different questions. Neither one substitutes for the other, and a report that doesn't say which one it's showing you should be treated as unverified.
Why did two internal reports on the same campaign show different reply rates?
An earlier progress report stated a figure whose exact denominator could no longer be reconstructed. A later platform export, isolated to the same campaign, with a reply count and a send count both visible and auditable, showed a different and much lower number (48 replies on 17,203 sends, versus a stated 1.65 percent on 17,175 sends). When that happens, the auditable figure is the one to use, not the one that's been repeated more often.
What's a good cold email reply rate in 2026?
Published benchmarks vary widely by methodology. Instantly's 2026 platform data puts the average at 3.43 percent with elite senders above 10 percent. Belkins' 2026 study, using a stricter replies-over-total-sends measure on net-new contacts, puts its 2025 average at 0.45 percent. Compare your own number to a benchmark that shares your denominator, not just the label "reply rate."
How do I check whether a reply-rate number I've been given is accurate?
Ask for the export, not the summary: a reply count and a send count from the same source and the same date range. If you can't get both numbers from one place, treat the percentage as unverified until you can.
What should a company do before publishing performance numbers about its own campaigns?
Reconstruct the denominator, source the number from an export rather than a prior summary, and treat any disagreement between two internal figures as a reporting problem to fix, not a discrepancy to average away.
