Find Duplicate Files with Python Without Deleting Anything

Downloads, project folders, and old backups have a habit of collecting extra copies. File names are not much help when one copy is called report-final.pdf and another is called report-final-2.pdf.

Here is a small Python tool that looks at file contents instead. It reports files with the same size and SHA-256 fingerprint, then leaves every decision to you. It has no delete, move, or rewrite command.

What you need

• Python 3.12 or newer. Check with python3 --version on macOS or Linux, or py -3 --version on Windows.
• A plain-text editor and a terminal open in the folder where you save the scripts.
• A small, local folder to inspect. Start with the practice files below, not your entire drive.

No additional packages or virtual environment are needed. The code uses Python's standard library. If Python is missing, use the installer and instructions at Python's official downloads page.

Tested environment: Linux with Python 3.12.14. Windows and macOS commands are included, but those platforms were not run for this article.

1. Save the finder

Create a plain-text file named find_duplicates.py and paste in the complete script below. Make sure your editor does not add a second extension such as .txt.

"""Report duplicate candidates. Never delete, move, or rewrite scanned files."""
import argparse
from collections import defaultdict
import hashlib
import os
from pathlib import Path
import stat
import sys

CHUNK = 1024 * 1024  # Read at most 1 MiB at a time.


def linked(path):
    return path.is_symlink() or path.is_junction()


def signature(info):
    return (info.st_dev, info.st_ino, info.st_size,
            info.st_mtime_ns, info.st_ctime_ns)


def digest_file(path, expected):
    before = path.lstat()
    if linked(path) or not stat.S_ISREG(before.st_mode):
        raise OSError("not an ordinary file")
    if signature(before) != expected:
        raise OSError("file changed since the scan")
    flags = os.O_RDONLY | getattr(os, "O_BINARY", 0)
    flags |= getattr(os, "O_NOFOLLOW", 0)
    with os.fdopen(os.open(path, flags), "rb") as stream:
        if signature(os.fstat(stream.fileno())) != expected:
            raise OSError("file changed before reading")
        digest = hashlib.sha256()
        while chunk := stream.read(CHUNK):
            digest.update(chunk)
        if signature(os.fstat(stream.fileno())) != expected:
            raise OSError("file changed while reading")
    if signature(path.lstat()) != expected:
        raise OSError("file changed after reading")
    return digest.hexdigest()


def scan(root):
    by_size = defaultdict(list)
    warnings = []

    def warn(error):
        warnings.append(str(error))

    for folder, directories, names in os.walk(
            root, topdown=True, followlinks=False, onerror=warn):
        safe = []
        for name in sorted(directories):
            path = Path(folder) / name
            try:
                if not linked(path):
                    safe.append(name)
            except OSError as error:
                warn(error)
        directories[:] = safe
        for name in sorted(names):
            path = Path(folder) / name
            try:
                info = path.lstat()
                if not linked(path) and stat.S_ISREG(info.st_mode):
                    by_size[info.st_size].append((path, signature(info)))
            except OSError as error:
                warn(error)

    groups = []
    hashed = 0
    for size, entries in sorted(by_size.items()):
        if len(entries) < 2:
            continue  # Different sizes cannot contain identical bytes.
        by_hash = defaultdict(list)
        for path, expected in entries:
            try:
                by_hash[digest_file(path, expected)].append(path)
                hashed += 1
            except OSError as error:
                warn(f"{path!s}: {error}")
        for digest, paths in sorted(by_hash.items()):
            if len(paths) > 1:
                groups.append((size, digest, paths))
    return groups, hashed, warnings


def main():
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("folder", help="an ordinary folder to inspect")
    args = parser.parse_args()
    if sys.version_info < (3, 12):
        parser.error("use Python 3.12 or newer")
    root = Path(args.folder).expanduser().absolute()
    try:
        if any(linked(p) for p in (root, *root.parents)):
            parser.error("choose a path without symbolic links or junctions")
        if not root.is_dir():
            parser.error("folder does not exist or is not a directory")
    except OSError as error:
        parser.error(str(error))

    groups, hashed, warnings = scan(root)
    for number, (size, digest, paths) in enumerate(groups, 1):
        print(f"\nGroup {number}: {len(paths)} paths, {size} bytes each")
        print(f"SHA-256: {digest}")
        for path in paths:
            print(f"  {str(path.relative_to(root))!r}")
    print(f"\n{len(groups)} candidate group(s); {hashed} file(s) hashed.")
    print("No files deleted, moved, or rewritten.")
    for warning in warnings:
        print(f"WARNING: {warning}", file=sys.stderr)
    if warnings:
        print("Scan incomplete: review warnings before relying on results.",
              file=sys.stderr)
    return 1 if warnings else 0


if __name__ == "__main__":
    raise SystemExit(main())

2. Make a harmless practice folder

Save the following as make_demo.py beside the finder. This separate setup script creates four practice files. It stops if duplicate-demo already exists, rather than overwriting an earlier exercise.

from pathlib import Path

root = Path("duplicate-demo")
root.mkdir()  # Stop rather than overwrite an existing demo folder.
(root / "subfolder").mkdir()
(root / "original.txt").write_bytes(b"hello\n")
(root / "subfolder" / "copy.txt").write_bytes(b"hello\n")
(root / "different.txt").write_bytes(b"world\n")
(root / "unique.txt").write_bytes(b"only one of this size\n")
print("Created duplicate-demo with four test files.")

On macOS or Linux, run:

python3 make_demo.py
python3 find_duplicates.py "duplicate-demo"

On Windows, run the same scripts with the Python launcher:

py -3 make_demo.py
py -3 find_duplicates.py "duplicate-demo"

3. Read the result

The tested Linux demo produced:

Group 1: 2 paths, 6 bytes each
SHA-256: 5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03
  'original.txt'
  'subfolder/copy.txt'

1 candidate group(s); 3 file(s) hashed.
No files deleted, moved, or rewritten.

original.txt and subfolder/copy.txt contain the same six bytes. different.txt is also six bytes long, but has different contents, so it does not join the group. unique.txt has a size no other file shares, so the script does not need to hash it. Windows will show backslashes in the relative paths.

The quoted paths make spaces and unusual characters easier to distinguish. Each group includes the full SHA-256 fingerprint, the number of matching paths, and the size of each file.

4. Try a real folder

Replace the example with a folder you own. Keep quotes around paths containing spaces.

python3 find_duplicates.py "/home/yourname/Documents/Old Downloads"

Or on Windows:

py -3 find_duplicates.py "C:\Users\YourName\Documents\Old Downloads"

Choose a quiet folder and let downloads, syncing, or editing finish first. Scanning large files can take time. Reading cloud-placeholder files may trigger downloads, so begin with files already stored locally. Press Ctrl+C if you need to stop.

How the script works

First, it walks through ordinary files and groups their paths by byte size. Only size groups containing at least two files move to the next step. It then reads those files in 1 MiB chunks, feeding each chunk to SHA-256. This keeps the file-reading buffer small even when a file is large, although the list of paths still takes memory.

Matching size and fingerprint produce a duplicate candidate. The script does not compare every byte between matches, and a hash match is not a mathematical guarantee. It also cannot tell you whether a copy is needed by another application.

Symbolic links and Windows junctions found during the scan are skipped. A starting path containing either is rejected. Other special files, such as named pipes, are ignored. The script checks file metadata around hashing and reports detected changes or read failures instead of silently treating those files as successfully checked.

These checks are intended for ordinary folders you control, not for hostile or actively changing directory trees. A path could change between checks, and a file could change after its hash is calculated. Stop other writers and rerun if the results matter.

Before you remove anything

Open the reported files, check where they are used, and make sure you have a backup before deciding what to keep. Identical contents can still be intentional: a project may need its own copy, and a backup is supposed to duplicate data.

Hard links can appear as separate matching paths while referring to the same underlying file. Empty files can match too. This tool does not calculate reclaimable disk space, inspect archive contents, or recognize visually similar photos saved with different compression. It reads file contents, not a promise that two paths are interchangeable.

Troubleshooting

• “Use Python 3.12 or newer”: Check the interpreter version. The junction check requires Python 3.12 or later.
• “Folder does not exist”: Check spelling, quotes, and your terminal's current folder. You can use a complete path instead.
• Symbolic-link or junction warning: Choose the actual folder path rather than a linked shortcut. A parent folder can be a link too.
• Permission or file-changed warning: The scan is incomplete. Close the application changing the file, or pick a folder your normal account can read, then rerun. Do not elevate permissions just to force a scan.
• No candidate groups: No matching size-and-hash groups were found among the files successfully checked. Review warnings before treating that as a complete answer.
• The demo folder already exists: Inspect it first. Rename the old practice folder or remove only that known practice folder in your file manager before rerunning the setup script.

For anyone using this in another script: exit status 0 means the scan completed without reported errors, 1 means warnings made the scan incomplete, and 2 means the command or starting folder was invalid. Finding duplicates is not an error.

Clean up

When you are done practicing, use your file manager to remove the duplicate-demo folder you created. Keep find_duplicates.py if it is useful, and remove make_demo.py if you no longer need the exercise. The finder creates no report files of its own and does not alter file contents; the operating system may still update access metadata when files are read.

Sources

• Python documentation: hashlib and incremental SHA-256 hashing
• Python documentation: directory walking, errors, and symbolic links
• Python documentation: path checks and Windows junctions