Deduplicate Records With Confidence Rules
Identifies and collapses duplicate or near-duplicate rows using deterministic and fuzzy rules in SQL.
Prompt
ROLE: You are a data engineer who deduplicates records while preserving the right surviving row. CONTEXT: Table [TABLE_NAME] (schema [SCHEMA]) contains duplicates. A true duplicate is defined by [DUP_KEY] (exact match) and/or near-match on [FUZZY_FIELDS] (e.g., normalized name + email). The surviving record should be the [SURVIVOR_RULE] (e.g., most recently updated, most complete). Engine: [DATABASE_ENGINE]. TASK: 1. Separate exact duplicates from near-duplicates and state the matching rule for each. 2. For exact dupes, write SQL using ROW_NUMBER() partitioned by [DUP_KEY], ordered by the survivor rule, keeping rn = 1. 3. For fuzzy dupes, normalize fields (lower, trim, strip punctuation) and group on the normalized key; note where similarity functions (LEVENSHTEIN/SOUNDEX/JACCARD) are needed and the engine support. 4. Produce both a 'rows to keep' query and a 'rows flagged as duplicates' query for review before deletion. 5. Recommend a safe delete/merge procedure (audit table first). OUTPUT FORMAT: Matching rules -> Exact-dedup ```sql``` -> Fuzzy-dedup ```sql``` -> Keep vs flag queries -> Safe deletion procedure. CONSTRAINTS: Never hard-delete before producing a reviewable flagged set. Make the survivor rule deterministic (add a tiebreaker so rn=1 is unique). Normalize before comparing. State the false-merge risk of fuzzy matching.
How to use this prompt
- 1
Copy the prompt above and paste it into ChatGPT, Claude, or Gemini — or open it in the visual Studio to edit each part on a canvas and run it with your own key.
- 2
Replace any bracketed placeholders with your specifics. The more concrete your context and constraints, the sharper the result — see the 5-part prompt structure.
- 3
Run it, then refine. Ask the model to critique and improve its own answer with self-critique prompting.
Techniques in this prompt
Assigns the model an expert persona so it adopts the right vocabulary, depth, and standards for the task.
Learn this techniqueForces explicit intermediate reasoning instead of jumping to a conclusion, which improves accuracy on hard tasks.
Learn this techniquePins the response to a defined structure so it drops straight into your workflow.
Learn this techniqueRecommended models
Build on this prompt
Open it in the visual Studio to wire it into a full workflow with your own API key — or learn the craft behind prompts like this.
More in Data Analysis & SQL
Translate Business Questions Into SQL
Turns a plain-English stakeholder question into a correct, well-commented SQL query against a known schema.
Optimize A Slow SQL Query
Diagnoses why a query is slow and rewrites it with targeted, explained optimizations and an index plan.
Debug A SQL Query That Returns Wrong Results
Systematically finds the logic error producing incorrect numbers and delivers a corrected, verified query.
Explain An Unfamiliar SQL Query In Plain English
Reverse-engineers a complex inherited query into a clear narrative, business meaning, and risk list.