CrapCleaner

Purpose

CrapCleaner is a simple Node.js console application for finding duplicate files inside a directory tree. It scans every file recursively, computes content hashes, and groups files with identical content so you can identify duplicate files and reclaim wasted storage.

Usage

1. Install Node.js if it is not already installed.
2. From the project root, run:

   npm start

3. When prompted, enter the path of the directory you want to scan.
4. The app will print a progress indicator while scanning and then write an HTML report to the system temporary directory.
5. The report will automatically open in your default browser.
6. To omit known system trash directories during a scan, pass one of the supported flags:

   npm start -- --exclude-trash
   # or
   npm start -- --exclude-system-trash

Example

npm start
# Enter directory to scan for duplicates: /home/user/Documents

The application will show the total number of files, a live progress message during the scan, and then open a browser report listing duplicate groups, the top 10 largest files, and wasted space. You can also pass --exclude-trash to omit known system trash folders from the scan.

Technical Details

How it works

1. Directory traversal

   • The app recursively walks the input directory using fs.promises.readdir with the withFileTypes option.
   • It visits every subdirectory and every regular file.

2. File counting

   • Before hashing, the app does one pass to count the total number of files. This enables a progress indicator during the main scan.

3. Hash generation

   • For each file, the app reads the file contents as a stream and computes a SHA-256 hash using Node's crypto module.
   • The hash is used as the dictionary key representing the file contents.

4. Duplicate grouping

   • A Map stores entries keyed by hash.
   • Each entry contains the file size and a list of file paths that produced the same hash.
   • If multiple files share the same hash, they are considered duplicates.

5. Duplicate ranking

   • Duplicate groups are ranked by size * (number_of_files - 1).
   • This value estimates the total bytes wasted by duplicate copies beyond one unique instance.
   • The app reports the top 100 groups by this duplicate value.

Implementation notes

• The app uses a progress indicator that updates the terminal line with the current file count, total count, and percentage complete.
• It handles file read errors gracefully by printing a skip message and continuing the scan.

License

This project is released under the MIT License.

MIT License

Copyright (c) 2026

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Vibe

This project was generated from a series of prompts to build a console application that finds duplicate files using recursive directory traversal, file hashing, and duplicate grouping. The structure, README, and implementation were produced with the help of an LLM model identified as Raptor mini.

All prompts used during this project are recorded in this section so development history and intent remain transparent.

Exact prompts used:

1. "This console application should ask the user for a directory as input and then loop over all the files in the directory recursively and make a hash from the file contents and store all files into a dictionary (key is the hash, value is a list of files with that hash, and a value for the size of the file in bytes). The goal is to find duplicate files accross the whole directory.

After the dictionary is created. We should find all the files that a are duplicates by seeing values in the dictonary have more than one file path. Report the top 100 duplicates by multiplying the size of the file by one less than the number of paths."

2. "Add a progress indicator while the scan is running"

3. "Write a readme in this project describing the purpose of the project. Add documentation about how the product should be used. Add a technical section describing in detail how the algorithm works. Add a "vibe" section describing the propt that was used to generate this project and what llm model was used."

4. "Include the exact prompts in the Vibe section"

5. "Can you write the output into an html file in the system temp directory and have it automatically open in the default browser. The report should include a total wasted space based on the top 10, top 100 and total findings as well as a percent of the hard drive for each. Make the wasted space in a human readable format (like change to GB or MB where it makes sense). Update the readme appropriately"

6. "Keep track of the 10 largest files regardless of if there's duplicates. Add them to the top of the report. Update the readme with the new functionality and don't forget to add this prompt to the Vibe section"

7. "Add command line options to omit the system trash"