The Report That Counts One Listing Twice

[1276amaranth]

Our public New Zealand directory research has a counting problem. We collect several pages of results through a local proxy, but the same business sometimes appears on two pages when the directory changes its ordering. The final total looks larger than the number of distinct businesses. I do not think more IP rotation will solve a duplicate created by moving records. What would you use as the basis for a reliable count? Some entries share similar names, so deleting rows by business name alone could also remove separate branches that should remain in the report.

[amazinglindi]

Use a stable listing identifier when the directory provides one, and keep the original URL with each record. If no identifier exists, define a cautious matching rule using several fields, such as address and phone number, while leaving uncertain cases for review. Also record when each page was fetched, because a moving sort order can place one listing in different positions during the same run. A proxy handles the connection; it does not make the source data static. Report distinct records separately from total rows collected so duplicate handling is visible instead of hidden inside a final number.

[38digital]

I would keep the first trial small enough to inspect every repeated entry. That helps you learn whether duplicates come from pagination, sponsored placements, or genuinely separate branches. Write down the matching rule before applying it to the full dataset. The new zealand proxy page offers plenty of useful information about routing and session modes for the connection setup. Keep that setup stable while testing the collection logic. Preserve the raw rows so a disputed merge can be reversed later. Your final count should reflect a documented definition of a distinct listing rather than whichever total the first collection happened to produce.