Dedupe

Dedupe is a Python library for teams building their own structured-record matching workflow. Its documentation describes learning from human match examples. The repository licenses the library under MIT; distinguish that code from the separately hosted Dedupe.io service when comparing products.

The practical question is whether your team can own the matching process around the library. That includes preparing fields, supplying representative training decisions, checking candidate coverage, retaining evidence, and maintaining the workflow when source data changes. A library does not assign the source owner who must resolve an ambiguous identity.

A proposed pilot with known cases

Use a frozen roster and the duplicate-record review workflow. This proposed evaluation uses invented venue records: “North Hall” and “N Hall” share an address and phone, while “North Hall Annex” has a different street number and the same central booking contact. Preserve original keys and raw fields throughout.

Ask the owner to label clear matching and different pairs with supporting references. Keep the annex pair unresolved if the owner cannot determine whether it is separately bookable. Do not label an ambiguous pair as a match simply to provide more training examples. The training set should record what was decided, by whom, and under which entity definition.

Reserve known examples for evaluation that were not used to shape the matching process. Include missing addresses, common names, and shared contact details. A useful pilot reveals which cases deserve more review; it should not be described as successful merely because the training examples receive plausible scores.

Inspect candidate coverage

Blocking makes candidate selection inspectable. Dedupe’s comparison explanation describes narrowing comparisons through shared features. Check a known matching pair with a missing postcode to see whether your chosen candidate process still includes it. Exclusion from the queue cannot establish a nonmatch.

Keep candidate scores separate from owner decisions. Compare the actual source fields and conflicts before accepting identity. Preserve unresolved pairs in the output, and examine connected groups for contradictory decisions rather than silently clustering them into one record.

For two sources with unrelated identifiers, use record linkage across datasets. A reviewed crosswalk must retain both namespaces and its cardinality policy before an analyst can use it in a join. Deduplication and linkage share comparison work but have different downstream outputs.

Choose the ownership model

Dedupe fits teams that want code-level control and can support a Python workflow. Plan for environment maintenance, source transformations, versioned training labels, evaluation records, and a review interface or packet appropriate to the owner. Record those responsibilities before treating the absence of a software subscription as the absence of operating cost.

If the team instead needs an ongoing managed mastering process, compare Tamr. Evaluate both against the same identity definition and evidence packet. This profile does not claim a hosted interface, particular accuracy, or automatic safe merges for the library. Choose after showing that your implementation retains provenance, makes ambiguous cases visible, and produces a reversible proposal the source owner can approve.