Probabilistic Catalog Matching with Fuzzy Scoring

Exact-key joins on ISRC or UPC resolve the majority of catalog records, but every royalty pipeline eventually hits a residual population where identifiers are missing, mistyped by an aggregator, or simply absent from a DSP’s delivery feed. That residual population is where probabilistic matching takes over from the deterministic pass described in Cross-Platform Catalog Matching, and it is also where a badly tuned scoring function does the most financial damage — a false-positive merge silently reassigns payout to the wrong rights holder, while a false negative leaves earned revenue stuck in exception queues. Getting the scoring function, blocking strategy, and threshold calibration right is a precision engineering problem, not a one-line fuzzywuzzy call.

Fuzzy Matching Method Trade-offs

Each scoring method trades recall for precision differently, and the choice of blocking key determines how many candidate pairs the scorer even sees. The table below assumes a catalog of 5–50 million tracks and a nightly reconciliation window.

Method	Strength	Weakness	Compute Cost	Royalty Impact
Levenshtein edit distance	Catches single-character typos in titles	Penalizes word-order swaps and abbreviations heavily	Low (O(n·m) per pair)	Under-merges reordered titles, leaving valid streams unmatched and revenue parked in exception queues
Token sort ratio (RapidFuzz)	Order-independent, tolerant of “feat.” / bracket noise	Can over-merge distinct remixes with similar tokens	Low–medium	Reduces false negatives but requires a duration or ISRC-prefix guard to avoid merging distinct masters
Jaro-Winkler	Weights prefix similarity; strong on artist name variants	Weak on transposed multi-word titles	Low	Best for payee/artist name reconciliation; used alone on titles it overpays cover versions as originals
Blocking on normalized artist + duration bucket	Cuts candidate pairs by 2–3 orders of magnitude before scoring	Misses matches where duration metadata itself is wrong	Very low (indexing only)	Essential for keeping nightly runs tractable; a bad blocking key silently excludes true matches from ever being scored
Weighted composite score (title + artist + duration + territory)	Balances signal from multiple fields, tunable per catalog	Requires ongoing weight calibration against labeled data	Medium	Most accurate for production payout decisions, but an unvalidated weight vector can systematically favor one field’s noise
Single fixed threshold (e.g., 0.85)	Simple, auditable	Ignores that precision/recall trade-off differs by field combination	None	A threshold set too low auto-merges ambiguous pairs into the wrong payee; set too high, it floods manual review and delays payout

Prerequisites & Assumptions

The implementation below targets Python 3.11+, rapidfuzz>=3.9 for vectorized string scoring, polars>=0.20 for candidate-pair generation and blocking, and pydantic>=2.6 for the match-result contract. It assumes incoming records have already passed schema validation and that isrc, artist_name, track_title, duration_seconds, and territory_code fields are populated (possibly with noise) but that a deterministic ISRC match has already failed or is absent — this stage only runs on the exact-match miss population.

Step 1: Build blocking keys to bound the candidate space

Scoring every incoming record against the full catalog is quadratic and will not finish inside a nightly window. Blocking keys narrow candidates to a plausible subset before any expensive string comparison runs.

python

import polars as pl
import unicodedata
import re

def normalize_text(value: str) -> str:
    value = unicodedata.normalize("NFKD", value).encode("ascii", "ignore").decode()
    value = re.sub(r"[^a-z0-9 ]", "", value.lower())
    return re.sub(r"\s+", " ", value).strip()

def build_blocking_key(df: pl.DataFrame) -> pl.DataFrame:
    return df.with_columns([
        pl.col("artist_name").map_elements(normalize_text, return_dtype=pl.Utf8).alias("artist_norm"),
        pl.col("track_title").map_elements(normalize_text, return_dtype=pl.Utf8).alias("title_norm"),
        (pl.col("duration_seconds") // 5 * 5).alias("duration_bucket"),
    ]).with_columns(
        (pl.col("artist_norm").str.slice(0, 4) + "_" + pl.col("duration_bucket").cast(pl.Utf8)).alias("block_key")
    )

def generate_candidate_pairs(incoming: pl.DataFrame, catalog: pl.DataFrame) -> pl.DataFrame:
    incoming_b = build_blocking_key(incoming)
    catalog_b = build_blocking_key(catalog)
    return incoming_b.join(catalog_b, on="block_key", how="inner", suffix="_catalog")

Step 2: Score candidate pairs with a weighted composite

Within each candidate block, apply per-field scorers and combine them into a single confidence value rather than trusting any one metric in isolation.

python

from rapidfuzz import fuzz
from pydantic import BaseModel, Field

class MatchScore(BaseModel):
    isrc: str | None = None
    catalog_isrc: str = Field(...)
    title_score: float = Field(ge=0, le=100)
    artist_score: float = Field(ge=0, le=100)
    duration_delta: int
    composite_score: float = Field(ge=0, le=100)
    confidence_tier: str

FIELD_WEIGHTS = {"title": 0.45, "artist": 0.40, "duration_penalty": 0.15}

def score_pair(row: dict) -> MatchScore:
    title_score = fuzz.token_sort_ratio(row["title_norm"], row["title_norm_catalog"])
    artist_score = fuzz.WRatio(row["artist_norm"], row["artist_norm_catalog"])
    duration_delta = abs(row["duration_seconds"] - row["duration_seconds_catalog"])
    duration_penalty = max(0.0, 100 - duration_delta * 8)

    composite = (
        title_score * FIELD_WEIGHTS["title"]
        + artist_score * FIELD_WEIGHTS["artist"]
        + duration_penalty * FIELD_WEIGHTS["duration_penalty"]
    )

    if composite >= 92:
        tier = "auto_merge"
    elif composite >= 78:
        tier = "manual_review"
    else:
        tier = "reject"

    return MatchScore(
        catalog_isrc=row["isrc_catalog"],
        title_score=title_score,
        artist_score=artist_score,
        duration_delta=duration_delta,
        composite_score=round(composite, 2),
        confidence_tier=tier,
    )

Step 3: Route by confidence tier, never by a single global cutoff

A single threshold treats “no ISRC at all” the same as “ISRC present but malformed,” which have very different risk profiles. Split routing so that low-risk auto-merges and high-risk manual reviews follow separate paths, and persist every score — not just the winner — for audit purposes.

python

def route_matches(scores: list[MatchScore], audit_sink) -> dict[str, list[MatchScore]]:
    routed: dict[str, list[MatchScore]] = {"auto_merge": [], "manual_review": [], "reject": []}
    for score in scores:
        routed[score.confidence_tier].append(score)
        audit_sink.write(score.model_dump())
    return routed

Verification & Validation

Before promoting a scoring change to production, replay it against a labeled holdout set of previously resolved matches and track the ISRC resolution rate (percentage of exact-match misses successfully closed by the fuzzy layer) alongside a precision figure computed from manually confirmed pairs. Assert that auto_merge precision stays above 99.5% — because auto-merges bypass human review, any drift below that threshold should block deployment. Log the full MatchScore payload, including rejected candidates, to an append-only table keyed by batch_id so that a royalty manager disputing a payout can reconstruct exactly why a track was or was not merged, and reconcile the count of auto_merge rows against the delta in the payout ledger for that reporting period.

Edge Cases & Gotchas

Cover versions and live re-recordings routinely score above 90 on title and artist alone because the composite weights in this example do not penalize duration mismatches heavily enough for short-duration catalogs — tune the duration penalty steeper for classical or spoken-word content where a 30-second difference is a different work entirely. Territory-code noise (US vs USA vs a missing code) will silently exclude legitimate blocking-key matches if normalization runs after the block key is built rather than before. Unicode artist names transliterated differently across DSPs (e.g., a Cyrillic or Hangul name romanized two different ways) will pass Jaro-Winkler poorly even when the underlying work is identical — maintain a transliteration lookup rather than relying on scoring alone. Finally, NULL duration fields break the bucket-based blocking key outright; coerce them to a sentinel bucket rather than letting the join silently drop the row.

Records that clear the manual_review tier but remain unresolved after human inspection should fall through to the Fallback Routing Logic Design layer rather than being force-merged, and persistent identifier collisions surfaced by low composite scores are often better resolved with the dedicated ISRC Collision Resolution Strategies covered separately.