Public-record business-data pipeline

I designed and built the pipeline end to end — ingestion, normalization, linkage, enrichment, and export — and the evaluation work around it.

1. Decision 01

   Layered, immutable processing

   Raw files are kept unchanged in dated snapshots. Each run moves through staging, normalization, identity, golden-record, and export layers, so every output can be traced to its source and rebuilt.

2. Decision 02

   Configuration over code

   Sources and column mappings live in configuration. Adding a source means a mapping and a small adapter, while every downstream step stays on one canonical schema.

3. Decision 03

   Columnar tools that fit in memory

   Polars and DuckDB over Parquet keep large joins and aggregations practical on a laptop, and streaming avoids decompressing large archives to disk.

4. Decision 04

   Deterministic links first

   Where records share an identifier they link on it directly. Only the remainder goes to probabilistic linkage, and records that stay ambiguous are kept as provisional entities rather than silently merged or dropped.

5. Decision 05

   Address enrichment with caching

   Addresses are geocoded through a public batch service with a distinct-address cache, so reruns and new sources only pay for addresses not seen before.

6. Decision 06

   Provenance on every field

   Each output field records which source supplied it, so coverage and disagreement can be measured per source.

Evaluation focused on whether links are right, whether fields agree with an independent reference, and whether the data is internally consistent.

Known-correct pairs for linkage

Some records carry the registry identifier of the business they refer to. Those give a set of known-correct matches to tune and test the linker against, without hand labelling.

Why stale addresses matter

Filings record the address at the time of filing, so a business that has since moved disagrees on street, city, or postal code. Address penalties tuned for current addresses wrongly reject those true matches. A separate weight profile for filing-time addresses recovered them without loosening the name requirements.

Why duplicate registry entries matter

Registries contain re-registrations and duplicate entities with identical names. They contaminate naive estimates of what a non-match looks like, and they create “wrong” links that are really duplicates. Measuring precision honestly meant auditing those cases separately.

Coverage and field agreement

Compared the output with a commercial reference dataset for coverage and agreement on overlapping fields, and examined which kinds of businesses each side is missing.

Data-quality checks

Automated checks for missing shares, dangling links between tables, and join keys that fail silently — for example identifiers exported with thousands separators.

• Pipeline and record linkage

  Layered processing, deterministic and probabilistic linkage, geocoding, provenance.

  Status: Local prototype

• Transparent public-data score

  An additive score with reason codes, built from public evidence and calibrated per state to a reference scale on held-out records. Filing-history features — how recently and how often a business files — mattered most where annual reports are not required.

  Status: Research

• Industry classifier

  Naive Bayes on business-name tokens with abstention, so it predicts only when confident. Self-training on confident predictions extended coverage; passes were compared at matched precision on a fixed holdout.

  Status: Research

• Python
• Polars
• DuckDB
• Parquet
• pytest
• rapidfuzz
• YAML
• Census Geocoder