Parsing DDEX ERN 4.2 Sales Messages with lxml

DDEX ERN 4.2 messages arrive in two shapes that a royalty pipeline must parse differently: a NewReleaseMessage, which describes catalog metadata through ReleaseList and ResourceList, and a SalesReportMessage, which reports consumption and payable events through DealTerms blocks tied back to those same resources. Both are namespace-qualified XML documents, and both routinely exceed hundreds of megabytes per delivery batch, which rules out a naive etree.parse() load into memory. This guide covers namespace-aware streaming extraction with lxml>=5 iterparse, how to pull DealTerms, ResourceList, and ReleaseList elements without loading the full DOM, and how to map the result into normalized rows for the reconciliation layer described in the DDEX ERN 4.2 Implementation Guide.

Parser Design Trade-offs

The choice of parsing strategy and element-handling discipline directly affects whether a malformed or partial delivery corrupts a payout run.

Parser Choice	Behavior	Memory Profile	Failure Isolation	Royalty Impact
`etree.parse()` full DOM	Loads entire document tree before any extraction	High — scales linearly with file size, unsuitable above ~200MB	Whole-file failure on any malformed node	A single bad `DealTerms` block anywhere in the file can abort the entire batch, delaying every payee in that delivery
`iterparse` with explicit namespace map	Streams `start`/`end` events, resolves tags via a pinned `{namespace}: prefix` dict	Low — only the active subtree is held in memory	Per-element; a malformed node can be logged and skipped	Isolated failures don’t block unrelated resources, so most of the batch still reaches the ledger on time
`iterparse` without `elem.clear()`	Streams events but never releases parsed nodes	Grows unbounded — effectively a memory leak on large files	Same as above until the process is OOM-killed	Batch runs that appear healthy in testing (small files) fail silently in production on full-size deliveries
Wildcard/local-name tag matching (`tag.endswith('DealTerms')`)	Avoids explicit namespace declarations	Low, same as pinned map	Silently matches unrelated elements if DDEX revises namespaces	Cross-version namespace drift can misroute a `DealTerms` block from a future ERN revision into current-period splits without raising an error
Two-pass parse (build resource index, then stream deals)	First pass indexes `ResourceList`/`ReleaseList` by ID, second pass streams `DealTerms` against that index	Moderate — index is small relative to the full document	High — deals referencing unknown resource IDs are caught explicitly	Prevents orphaned deals (a `DealTerms` block pointing at a `ResourceReference` that doesn’t exist) from ever reaching the payout table

The two-pass, pinned-namespace iterparse approach with explicit elem.clear() calls is the only combination that scales to production file sizes while still failing safely on malformed input.

Prerequisites & Assumptions

This guide assumes Python 3.11+, lxml>=5.0 (which stabilized the iterparse events API against namespace-qualified tags), and pydantic>=2.4 for the normalized output model. It assumes ERN 4.2 messages validated against the DDEX XSD have already passed the schema-validation stage described in Validating DDEX XML Against XSD Schemas — this guide covers extraction and mapping, not XSD conformance. Sample files are assumed to declare the standard ern namespace (http://ddex.net/xml/ern/42) and the shared AVS/MessageHeader namespaces used across ERN 4.x releases.

Implementation

1. Namespace-Aware Streaming Setup

Pin the namespace map once rather than resolving it per-element, and iterate only on the events you need.

python

from lxml import etree

ERN_NS = {"ern": "http://ddex.net/xml/ern/42"}


def iter_ern_elements(file_path: str, local_names: set[str]):
    qualified_tags = tuple(f"{{{ERN_NS['ern']}}}{name}" for name in local_names)
    context = etree.iterparse(
        file_path, events=("end",), tag=qualified_tags, huge_tree=True
    )
    for _, elem in context:
        yield elem
        elem.clear()
        while elem.getprevious() is not None:
            del elem.getparent()[0]

2. Building the Resource and Release Index (Pass One)

Index SoundRecording resources and Release records by their DDEX-internal reference IDs before touching any DealTerms, so deal-to-resource joins can be validated rather than assumed.

python

from pydantic import BaseModel


class ResourceRecord(BaseModel):
    resource_reference: str
    isrc: str | None = None
    title: str | None = None


class ReleaseRecord(BaseModel):
    release_reference: str
    upc: str | None = None
    release_title: str | None = None


def build_resource_index(file_path: str) -> dict[str, ResourceRecord]:
    index: dict[str, ResourceRecord] = {}
    for elem in iter_ern_elements(file_path, {"SoundRecording"}):
        ref = elem.findtext("ern:ResourceReference", namespaces=ERN_NS)
        isrc = elem.findtext(".//ern:ISRC", namespaces=ERN_NS)
        title = elem.findtext(".//ern:TitleText", namespaces=ERN_NS)
        if ref:
            index[ref] = ResourceRecord(resource_reference=ref, isrc=isrc, title=title)
    return index


def build_release_index(file_path: str) -> dict[str, ReleaseRecord]:
    index: dict[str, ReleaseRecord] = {}
    for elem in iter_ern_elements(file_path, {"Release"}):
        ref = elem.findtext("ern:ReleaseReference", namespaces=ERN_NS)
        upc = elem.findtext(".//ern:ICPN", namespaces=ERN_NS)
        title = elem.findtext(".//ern:TitleText", namespaces=ERN_NS)
        if ref:
            index[ref] = ReleaseRecord(release_reference=ref, upc=upc, release_title=title)
    return index

3. Streaming DealTerms Against the Index (Pass Two)

Every DealTerms block references a ResourceReference or ReleaseReference. Validate the reference against the pass-one index before emitting a normalized row, and quarantine anything that fails to resolve rather than dropping it silently.

python

class NormalizedDealRow(BaseModel):
    batch_id: str
    resource_reference: str
    isrc: str | None
    territory_code: str
    right_share_pct: float
    reporting_period: str


def extract_normalized_deals(
    file_path: str,
    resource_index: dict[str, ResourceRecord],
    batch_id: str,
) -> tuple[list[NormalizedDealRow], list[dict]]:
    rows: list[NormalizedDealRow] = []
    quarantined: list[dict] = []

    for elem in iter_ern_elements(file_path, {"DealTerms"}):
        ref = elem.findtext(".//ern:ResourceReference", namespaces=ERN_NS)
        territory = elem.findtext(".//ern:TerritoryCode", namespaces=ERN_NS)
        share_text = elem.findtext(".//ern:RightSharePercentage", namespaces=ERN_NS)
        period = elem.findtext(".//ern:ReportingPeriod", namespaces=ERN_NS)

        resource = resource_index.get(ref) if ref else None
        if resource is None or territory is None or share_text is None:
            quarantined.append({"batch_id": batch_id, "raw_reference": ref, "reason": "unresolved_or_incomplete"})
            continue

        rows.append(
            NormalizedDealRow(
                batch_id=batch_id,
                resource_reference=ref,
                isrc=resource.isrc,
                territory_code=territory,
                right_share_pct=float(share_text),
                reporting_period=period or "UNKNOWN",
            )
        )
    return rows, quarantined

Verification & Validation

After extraction, assert that the count of DealTerms elements found by iterparse equals the count of rows plus quarantined records — a mismatch means an exception was swallowed silently somewhere in the traversal. Sum right_share_pct per resource_reference and territory_code pair and confirm it does not exceed 100.00% before the rows reach the split-calculation stage; a sum above that threshold indicates either a duplicated DealTerms block or a resource-index collision from pass one. Cross-check that every isrc surfaced in the normalized rows is well-formed before it enters the ISRC-to-ISWC Mapping Workflows stage, and record the ratio of resolved-to-quarantined DealTerms blocks per batch as a standing SLA metric — a healthy production feed should quarantine well under 1% of deals.

Edge Cases & Gotchas

ERN namespace drift is the most common silent failure: a delivery declaring ern/43 instead of ern/42 will not match a hardcoded ern/42 tag filter, and iterparse will simply yield zero elements rather than raising an error, so always assert a non-zero element count after the first pass. Watch for DealTerms blocks that reference a ReleaseReference instead of a ResourceReference (valid for release-level deals but easy to miss if your index only covers resources) — build both indexes even if a given batch appears to use only one. Some senders omit ReportingPeriod at the deal level and place it only in the message header; fall back to MessageHeader/MessageCreatedDateTime rather than defaulting to the current date, since a wrong reporting period silently shifts revenue into the wrong payout cycle. Finally, huge_tree=True disables some of lxml’s entity-expansion safety limits, so only enable it for deliveries received over an authenticated, trusted DSP channel.

Extraction choices here are strictly about DDEX XML parsing; if a delivery instead arrives as a flat sales file, the ingestion trade-offs are covered separately in Engineering DDEX vs CSV Metadata Ingestion. Both approaches ultimately feed the same normalized schema described in the DDEX ERN 4.2 Implementation Guide.