Sizing a bloom filter from the row

How wide should a document's bloom be, and how many hash functions should it use? Start from how many pairs a row carries, not from the universe of keys.

The reasoning, in four steps

1

Start from the row's population

The knob is how many classification pairs a single row carries: its 99th percentile, measured per bloom facet. The universe of keys and values doesn't matter at all. That's the opposite of Prometheus or Loki, where every distinct combination of label values becomes its own series or stream and grows the index. A bloom's cost is fixed per row: a value seen once costs exactly what a value seen a million times costs. Only the per-row fill matters.

2

Choose the rate you can tolerate

A false positive costs one wasted fetch of a document, and a query scanning N rows expects about p × N of them. So the useful target is "at most f false fetches per query", which gives p = f / N. Common choices are one in a thousand and one in ten thousand.

3

The rate alone fixes k

The optimal number of hash functions is k = log₂(1/p): 10 for one in a thousand, 13 for one in ten thousand. It doesn't depend on the width, so changing the width never forces a change of k. And k must be a constant for each bloom facet, because a query hashes its criteria with the same k the stored blooms were built with. That's also why the population to size for is the 99th percentile, not the mean: tune k for typical rows and the large ones degrade sharply, while sizing for the 99th percentile gives typical rows far better rates than the target.

4

The width follows, in powers of two

For a given rate, the bits needed per pair are constant: about 14.4 at one in a thousand and 19.2 at one in ten thousand. So the population a width supports is simply the width divided by that, and each doubling of the width doubles it. Keeping widths to powers of two, from 128 to 1,024 bytes, turns sizing into a ladder rather than a calculation, and every rung fits the narrow row's page budget.

The ladder

The largest 99th-percentile population each width supports while keeping the false-positive rate at or below the target:

WidthBitsOne in 1,000One in 10,000
kp99 supportedkp99 supported
128 B1,02410711353
256 B2,0481014213106
512 B4,0961028413213
1,024 B8,1921056913427

The rule: choose the smallest power-of-two width whose supported p99 covers the measured p99, with k = 10 for one in a thousand or 13 for one in ten thousand. Rows above the p99 degrade gracefully, and verification of the actual document still removes every false positive before anything is returned.

Try it

A worked case. A measured p99 of 300 pairs at one in a thousand falls just above the 512-byte rung, which supports 284. Either move up to 1,024 bytes, good for 569, or keep 512 bytes and accept about 1.4 in a thousand at the p99, which is barely worse than the target.

For contrast: PostgreSQL's bloom index

PostgreSQL's bloom extension has the same two knobs under other names: length is the signature width in bits, and colN is the number of bits set per indexed column, a k per column. It can weight columns individually because its population is fixed: every row contributes exactly one value per indexed column. A document's population is variable and open-ended, so its k must be uniform per facet. That's the price of covering keys nobody anticipated. A facet of fixed, known fields could adopt per-field weighting; an open, custom facet cannot.

Computing the indices costs almost nothing even at high k: double hashing derives all k positions from two hash computations, so k is purely a false-positive decision, never a CPU one.