Diagnosing OOM During Large ERN Batch Runs

A multi-gigabyte ERN delivery from a major DSP will reliably kill a worker that parses it with lxml.etree.parse() or loads it wholesale into a pandas.DataFrame, because both approaches hold the fully materialized structure in memory at once on top of Python’s own per-object overhead. This page narrows in on that specific failure inside the broader troubleshooting ingestion pipeline failures reference: why DOM-style parsing runs out of headroom on large batches, and which streaming techniques keep peak memory bounded regardless of file size.

The core problem is that etree.parse() builds a full in-memory tree — every element, attribute, and text node as a Python/C object — before your code ever touches a single SalesTransaction. A 3 GB ERN file with millions of repeated transaction elements can easily balloon to 10-15x its on-disk size once parsed, because libxml2’s node objects and lxml’s Python wrappers both carry overhead well beyond the raw bytes. pandas.read_csv() without chunksize has the analogous problem for CSV sales reports: the entire frame, plus dtype-inference scratch space, must fit in RAM simultaneously.

Parsing Approach Trade-offs

Approach	Peak Memory	Complexity	Random Access	Royalty Impact
DOM parse (`etree.parse`)	Unbounded — scales with full file size	Low to write	Yes, full tree available	Highest OOM risk; a killed worker mid-batch can leave reconciliation tables partially updated, forcing a manual reconcile against the DSP statement
`iterparse` with element clearing	Bounded — one element subtree at a time	Moderate — must manage `elem.clear()` and sibling references	No — forward-only	Safest for large batches; predictable memory means batches complete deterministically, reducing partial-batch cleanup work
SAX (`xml.sax`)	Bounded — event callbacks only	Higher — manual state machine for nested elements	No — forward-only, no tree at all	Lowest memory footprint but easy to mis-track nesting state, which risks silently dropping a `right_share_pct` if a handler misses a close event
Streaming CSV (chunked `pandas`/Polars lazy scan)	Bounded — one chunk/batch at a time	Low with Polars `scan_csv`, moderate with pandas `chunksize`	No — sequential	Keeps aggregation correct as long as split-sum checks run per chunk and are re-aggregated, not skipped for convenience

For ERN specifically, iterparse is the pragmatic default: it gives you element-level access without SAX’s manual state tracking, and clearing processed elements keeps peak memory flat regardless of how many SalesTransaction records the file contains.

Prerequisites & Assumptions

The examples below assume Python 3.11+, lxml>=5.0, polars>=0.20, and pyarrow>=14 in the ingestion environment, running against ERN 4.2/4.3 sales message files that can range from hundreds of megabytes to several gigabytes, and CSV sales reports on the same order. Workers are assumed to run in memory-limited containers (commonly 1-4 GB) rather than unconstrained VMs, which is the actual constraint that makes streaming necessary rather than optional.

Implementation

Step 1: Replace DOM parsing with iterparse and explicit clearing

The key discipline with iterparse is clearing each element — and its preceding siblings — once you’ve extracted what you need, so libxml2 can reclaim the memory instead of holding the whole tree for the life of the parse.

python

from lxml import etree

ERN_NS = {"ern": "http://ddex.net/xml/ern/43"}

def stream_sales_transactions(file_path: str):
    """Yield parsed transaction dicts without ever holding the full ERN tree in memory."""
    context = etree.iterparse(
        file_path, events=("end",), tag="{http://ddex.net/xml/ern/43}SalesTransaction"
    )
    for _, elem in context:
        isrc = elem.findtext("ern:ISRC", namespaces=ERN_NS)
        territory_code = elem.findtext("ern:TerritoryCode", namespaces=ERN_NS)
        right_share_pct = elem.findtext("ern:RightSharePercentage", namespaces=ERN_NS)
        yield {
            "isrc": isrc,
            "territory_code": territory_code,
            "right_share_pct": float(right_share_pct) if right_share_pct else None,
        }
        # Free the element and any preceding siblings still held by the parent.
        elem.clear(keep_tail=True)
        while elem.getprevious() is not None:
            del elem.getparent()[0]

Step 2: Bound worker memory with chunked aggregation instead of full-frame loads

For CSV sales reports, replace a single read_csv call with a lazy scan that only materializes one chunk at a time, aggregating incrementally so peak memory tracks chunk size, not file size.

python

import polars as pl

def aggregate_royalties_streaming(csv_path: str, batch_id: str) -> pl.DataFrame:
    lazy_frame = pl.scan_csv(csv_path, schema_overrides={"right_share_pct": pl.Float64})
    aggregated = (
        lazy_frame.group_by(["isrc", "territory_code"])
        .agg(pl.col("right_share_pct").sum().alias("total_share_pct"))
        .filter(pl.col("total_share_pct").is_not_null())
    )
    return aggregated.collect(streaming=True)

Step 3: Spot the leak before it kills the worker

If memory still climbs steadily even with iterparse, the leak is usually a reference held outside the loop — a list accumulating every parsed record instead of writing them out incrementally. Confirm with tracemalloc against a resident-set-size sample:

python

import os
import tracemalloc

import psutil

def monitor_parse_memory(file_path: str, sample_every: int = 5000) -> None:
    tracemalloc.start()
    process = psutil.Process(os.getpid())
    for i, _ in enumerate(stream_sales_transactions(file_path)):
        if i % sample_every == 0 and i > 0:
            rss_mb = process.memory_info().rss / (1024 * 1024)
            current, peak = tracemalloc.get_traced_memory()
            print(f"record={i} rss_mb={rss_mb:.1f} traced_current_mb={current / 1e6:.1f} peak_mb={peak / 1e6:.1f}")
    tracemalloc.stop()

A healthy stream holds RSS roughly flat after an initial ramp-up. If RSS keeps climbing linearly with record count, look for accumulator lists, unclosed file handles, or a downstream consumer (such as a database writer) that batches writes far less often than the parser produces records.

Verification & Validation

After switching to streaming parsing, re-run the batch against a known file and confirm three things: peak RSS stays under the container limit (check via the monitor_parse_memory output or container cgroup stats), the record count matches the manifest exactly, and the aggregated right_share_pct sums still resolve to 100% per ISRC/territory pair. A streaming rewrite that silently drops the last partial chunk on EOF will pass a smoke test but fail this last check — always assert split-sum totals on the full run, not a sample.

Edge Cases & Gotchas

Namespace drift between ERN 4.2 and 4.3 profiles will make iterparse’s tag filter match nothing at all — a silently empty generator, not an exception — so verify the namespace URI against the actual file’s root element before assuming the parser is broken. Elements with lazy or deferred child population (some ERN generators emit SalesTransaction before its RightsController child is fully written in a streamed producer) can trip up elem.clear() if called too early; only clear after confirming end event semantics guarantee the subtree is complete, which they do for well-formed files but not for ones truncated mid-write. For CSV, chunked reads can split a multi-row logical batch (a track split across two contributor rows) across chunk boundaries if the file isn’t sorted by isrc first — sort or pre-partition upstream, or aggregate is applied before the check completes and the split-sum assertion above will falsely fail on a boundary that isn’t actually a data error.

The delimiter-drift and semantic-validation issues that often accompany large CSV batches are covered in troubleshooting ingestion pipeline failures, which also walks through diagnosing stuck cursors and duplicate-batch cleanup once a streaming rewrite is in place.