Back to Home
    SmartVisuals Research · Likert, Crosstab, NPS

    Methodology

    The formulas, conventions and degenerate cases behind every number the Research visuals put on screen. Written for the person who has to defend the numbers in a client meeting.

    Every number the SmartVisuals Research visuals put on screen is computed offline, inside the visual, by one shared statistics engine. This page documents exactly what that engine computes: the formulas, the conventions behind them, what happens in degenerate cases, and what version 1 deliberately does not do.

    It is written for the person who has to defend the numbers in a client meeting. If something here does not match what you would have done, the difference is stated rather than hidden — and we would like to hear about it: hello@smartvisuals.io.

    What the visual receives

    Power BI hands a visual aggregated data, not respondent rows: one value per Question × Answer × Segment cell. So the engine is built around three sums your model provides as measures:

    MeasureMeaning
    ValueΣw over the cell — or a plain distinct-respondent count when unweighted
    Sum of Weight²Σw² over the base — optional; enables weighted inference
    —the unweighted count, when you map a count measure

    Nothing else is needed, and nothing else is sent anywhere. Where the engine does receive respondent-level rows, it deduplicates by respondent before counting (first row wins), so a long-format table with one row per respondent × statement cannot inflate a base.

    Bases

    Three bases are reported, and they mean different things:

    • Unweighted base n — distinct respondents. This is what "n=" shows, and what low-base rules are applied to.
    • Weighted base Σw — the sum of weights; the denominator of every percentage.
    • Effective base n_eff = (Σw)² / Σw² — the Kish effective base. This is the base every standard error and every test uses.

    A missing or absent weight column means w = 1.0 for every respondent. In that case n_eff = n exactly, and every formula below reduces to its classical unweighted form. Weighting does not switch on a different set of statistics — it only changes the base the same statistics run on.

    Weights must be finite and non-negative; anything else is rejected rather than silently coerced.

    Why the effective base

    Weighting trades bias for variance: a sample with uneven weights carries less information than its weighted total suggests. Using Σw as the base of a significance test would treat that lost information as if it were real and overstate significance. n_eff is the standard correction — it is what SPSS CTABLES-style column tests use, and it is what an agency tab deck implies when it prints weighted percentages with letters.

    Percentages

    A weighted proportion is p = Σw(hits) / Σw(base), reported on the base you chose in the field wells. N/A answers can be kept in or taken out of the base; the footnote records which.

    Margin of error

    The visuals show ± next to headline figures and in tooltips. It is the half-width of the normal (Wald) interval on the effective base:

    MoE = z(α/2) · sqrt( p(1−p) / n_eff )

    z(α/2) is computed by bisection on the normal CDF and matches SciPy's norm.ppf to 1e-12. The interval itself matches statsmodels proportion_confint(method="normal").

    This is the Wald interval, chosen because it is the one research reports use and the one that stays consistent with the pooled z-test below. It is known to be optimistic for proportions very close to 0 or 1 on small bases; at the bases where that matters the low-base rules below have usually already flagged the cell.

    Significance letters between columns

    The letters (A/B/C…) are the core of a tab deck, and they are a two-sample pooled z-test on weighted proportions with effective bases:

    p̂ = (p₁·n₁ + p₂·n₂) / (n₁ + n₂)          where n are effective bases
z  = (p₁ − p₂) / sqrt( p̂(1−p̂) · (1/n₁ + 1/n₂) )

    Every ordered pair of columns is tested. A column's letters list the columns it is significantly higher than, at the confidence level you set (90 %, 95 % or 99 %). Columns are lettered A, B, C … Z, then AA, AB, …

    The same machinery, with the same letter and low-base conventions, produces:

    • segment letters in the Likert battery and the crosstab banner,
    • mean-row letters (via the Welch test below),
    • NPS segment letters and wave comparisons.

    Because one implementation drives all of them, a segment comparison and a wave comparison cannot disagree about the same two samples.

    Column overlap — a documented limitation

    Version 1 treats banner columns as independent samples. When the same respondent appears in two columns (overlapping banner definitions, multi-select segments), the independence assumption does not hold and the test is anti-conservative — it will call differences significant slightly more often than it should.

    This is the same assumption the majority of tab-deck tooling makes by default, but it is an assumption, so: if your banner columns overlap, read the letters as indicative. Dependent-sample tests for overlapping columns are planned for version 2.

    Multiple comparisons

    A banner with k testable columns runs k(k−1)/2 pairwise tests per row, so some letters will appear by chance alone. Two options:

    • None (default) — matches the dominant agency convention for lettered tab decks. The letters are pairwise statements, not a family-wise claim.
    • Bonferroni — α is divided by the number of comparisons actually performed for that row (low-base columns are excluded from the count, so the correction is not inflated by columns that were never tested).

    The default is "none" because that is what the deliverable convention expects, not because it is the more conservative choice. If you are making a decision off a single letter in a wide banner, turn Bonferroni on.

    Low bases

    A column whose unweighted base falls below the threshold (default 30) is:

    • flagged in the visual (hatching, an asterisk, or both),
    • excluded from every comparison — it neither receives nor gives letters,
    • excluded from the Bonferroni comparison count.

    The threshold is on the unweighted base deliberately: weighting can make a 28-respondent cell look like 60, and it is the 28 actual interviews that limit what can be said.

    Nets (top box, bottom box)

    A net is a derived proportion: the share of respondents whose answer falls in the net's scale positions. Scale order comes from your model's sort order — set the answer column's Sort By Column and the engine reads the scale the way you defined it, rather than guessing from labels.

    Because a net is just a proportion, it flows through exactly the same tests, letters, margins of error and low-base rules as any other row. There is no separate "net statistics" path.

    Standard definitions for an n-point scale take the top and bottom b positions, with b capped at floor(n/2) so top and bottom boxes can never overlap on a short scale.

    Means

    Scale means are weighted, with the frequency-weight variance:

    x̄  = Σwx / Σw
s² = Σw(x − x̄)² / (Σw − 1)

    s² is NaN when Σw ≤ 1 — there is no dispersion to estimate from one observation, and the engine says so instead of returning zero.

    Means from a frequency table (the shape a visual actually receives — a weighted count per scale point) use the same formulas; for weighted data Σw² has to come from the base, because bucket totals alone cannot recover it.

    Comparing means

    A Welch t-test on effective bases, with Welch–Satterthwaite degrees of freedom:

    SE² = s₁²/n₁ + s₂²/n₂
t   = (x̄₁ − x̄₂) / SE
df  = SE⁴ / ( (s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1) )

    Welch rather than Student because survey segments routinely differ in both size and variance, and the equal-variance assumption is the first thing to fail. With unit weights this is exactly the classical Welch test. The t CDF is implemented via the regularized incomplete beta function and matches SciPy to 1e-10.

    Comparing against a benchmark or norm

    A benchmark is a fixed reference value with no sampling error of its own — an industry norm, a target, a published figure. It is therefore a one-sample test, with the variance under the null taken from the reference proportion:

    z = (p − p₀) / sqrt( p₀(1−p₀) / n_eff )

    This matches statsmodels proportions_ztest(count, nobs, value=p₀, prop_var=p₀). If your "benchmark" is itself an estimate from a survey, this test understates the uncertainty — compare it as a segment instead.

    Tracker mode (wave over wave)

    Tracker deltas compare the current wave against the previous one using the two-sample test appropriate to the quantity: the pooled z-test for proportions and nets, the Welch test for means, the NPS difference test for scores. Waves are treated as independent samples, which is right for fresh-sample tracking and conservative for a panel measured repeatedly (where a paired test would have more power).

    NPS

    NPS = 100 · (promoter share − detractor share)

    Default coding is 0–6 detractor / 7–8 passive / 9–10 promoter, and the cut-offs are remappable if your organisation uses different ones. Shares are weighted; the base is the effective base.

    Standard error

    NPS is a difference of two shares of the same three-category multinomial, so its variance is not the sum of two independent proportion variances:

    Var = ( p_P + p_D − (p_P − p_D)² ) / n_eff
SE  = 100 · sqrt(Var)                        on the −100..100 scale

    The ± on the headline score and the confidence band on the trend line are z(α/2) · SE.

    Change and movement

    A wave-over-wave change, or one segment against another, is a two-sided z-test on the difference of two independent NPS estimates:

    z = (NPS₁ − NPS₂) / sqrt( SE₁² + SE₂² )

    "A change this size would need n ≈ …"

    When a change is not significant, the visual can say how large a wave it would have taken. With v = p_P + p_D − (p_P − p_D)² averaged over the two samples and two equal independent samples of size n, the difference has SE = 100·sqrt(2v/n), so:

    n = 2v · ( 100 · z(α/2) / |d| )²

    This is a sample-size hint for planning the next wave. It is not a verdict on the current one.

    Gap to target

    How many respondents would have to move for the score to reach a target, at the current base: one point of NPS per 1 % of the base, so respondents = ceil((target − NPS) / 100 · Σw), and zero once the target is met. It assumes movement from passives — promoters gained from passives, or detractors moved to passives — which is the cheaper of the two routes and therefore the optimistic one.

    Degenerate cases

    The engine does not fabricate a number it cannot compute. Every test returns NaN — and the visual shows nothing rather than a marker — when:

    • either base is empty, or the effective base is not greater than 1 for a mean test;
    • both proportions are 0, or both are 1 (no evidence of a difference exists, and the pooled standard error is zero);
    • a variance is zero or undefined, so the Welch statistic has no denominator;
    • an NPS estimate is degenerate (no valid scores in the base).

    A missing letter therefore means "not tested or not testable", never "tested and equal". Where the distinction matters to a reader, the footer says so.

    What version 1 does not do

    Stated plainly, because the absence of a statistic is part of a methodology:

    • Dependent/overlapping column tests — see the caveat above. Version 2.
    • Chi-square block tests — no overall test of association per crosstab block; the cell letters carry the practical value first. Version 2.
    • Design effects beyond weighting — clustering and stratification are not modelled. n_eff corrects for unequal weights, not for a complex sample design. If your study has a clustered design, the letters are anti-conservative.
    • Finite population correction — not applied.
    • Imputation — none. Missing answers are missing; you choose whether N/A codes sit in or out of the base.
    • MaxDiff, conjoint, verbatim coding — out of scope.

    Validation

    The engine is plain TypeScript with no dependencies, including for the distributions: the normal and t CDFs are implemented locally (Cody-style rational approximations and the regularized incomplete beta function) so that nothing has to be fetched at runtime.

    Correctness is established three ways, and every release has to pass all of them:

    1. Golden fixtures generated by SciPy/statsmodels, diffed against the engine at 1e-9 for estimates and bases and 1e-8 for p-values, plus an independent re-check of every fixture in R (pnorm, pt, pbeta, survey::svymean) at 1e-8.
    2. Varied-dataset validation: two complete synthetic studies with deliberately awkward structures — rim-style weights versus no weight column at all, 5- and 7-point scales, explicit N/A codes, scattered missing answers, duplicate respondent rows, pooled low-base segments (n = 28 and n = 8) and constant-answer cells with zero variance — tabulated end to end through the public API and diffed cell by cell, base by base, letter by letter and p-value by p-value against a SciPy/statsmodels reference, then cross-checked a third time in R (survey::svymean, t.test, prop.test).
    3. Property tests: weights of 1.0 must reproduce unweighted results exactly; letters must be antisymmetric (A > B implies not B > A); bases must stay invariant under filtering.

    Latest run: 2026-08-31 — golden fixtures all passed, engine == SciPy == R on both datasets (13/13 engine tests, 90/90 R checks). Environment: Node 22, SciPy 1.17.1, statsmodels 0.15.0, R 4.3.3 with survey 4.2.1.

    What this does not cover: a diff against a production SPSS tab deck from a real fielded study. We would welcome one — if you have a study you can share, get in touch.

    No external services

    Every statistic on this page is computed inside the visual, in your browser or your copy of Power BI Desktop. The visuals make no HTTP, WebSocket or any other outbound request; they carry no telemetry, load no external fonts or scripts, and write nothing to browser storage. No report data leaves Power BI — which is also why the statistics cannot be shortcut by a cloud API, and why the visuals are eligible for Microsoft's Power BI certification.

    Licensing is resolved by the Power BI host through the native License Manager, so even the entitlement check makes no call of its own.

    Questions, disagreements or a correction: hello@smartvisuals.io — we read every message. This page is versioned alongside the engine; the conventions above describe version 1.