Universal Text Conversion ✨

A repository for handling virtually all types unstructured text documents and converting them into docx or pdfs. Essentially parses through several nested folders (zipped or otherwise) to access all the text files and convert them to pdf. Used for the Department of Organization Studies.

⚡ Features

1. Automatic Zip Extraction
   • Handles top-level and nested ZIP archives.
2. Combining Different Formats
   • Merge DOC (1997-2003), DOCX, RTF, ODT, PDF, and TXT files together seamlessly.
3. Large File Splitting
   • Splits final PDFs if page counts exceed a chosen threshold, to keep them more manageable.
4. Ignore Irrelevant Folders and Corrupt Files
   • Skips merging text from specified folders (e.g., literature, __MACOSX) and skips merging corrupt docx/pdf files.

For Returning Users:

Assuming you already have the repository and the environment on your computer, you need to do the following:

1. Open Windows command window and navigate to your project folder using cd path/to/project and:

git pull

2. Activate and sync the environment

.venv\Scripts\activate
uv pip sync requirements.txt

🔧 Installation

1. Install Python

Make sure you have a suitable version of Python 3.x installed.
Any version after 3.6 recommended.

2. Install uv.

uv is a python dependency manager. To install it, open the Windows Powershell command window on your windows machine using the search option in your task bar. Inside the powershell window, paste the following line:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

3. Install Git

If you do not have Git installed, follow these steps:

• Windows: Download and run the installer from git-scm.com.

4. Install LibreOffice

Go to this link and download libre office. Please make sure you are installing it in the C drive. The standard location is C:\Program Files\LibreOffice.

Note: If you are choosing to download it elsewhere (or it happens to be located elsewhere), please locate the path of the soffice.exe file. It is located inside the LibreOffice\program folder. Once you have located the soffice.exe file, please replace the DEFAULT_SOFFICE variable at the top of the conversion_utils.py script.

🚀 Usage:

Go to a desired folder location in your PC and open git bash. This can be done by right clicking and selecting open git bash here.

1. Clone the Repository

   git clone https://gitlab.uvt.nl/tsb-rst/universal_text_conversion.git

You can now close the git bash terminal.

2. Create the uv environment

Next, open the windows terminal (search for Command Prompt) from the start menu and navigate to your project folder chosen in step 1. Then, we can create uv environment. This step will use the requirements.txt file and also install all the dependences. This can be done using the code segments below.

1. Navigating to your project folder

cd path/to/your/project/folder

2. Making a uv environment.

Initializing the environment and downloading the dependencies

uv sync 

Activating the environment (This is done every time you wish to run the script!)

.venv\Scripts\activate

3. To run the merging script, first activate your environment (if you haven't already) and then write the code below in the windows terminal. The repository comes with a dummy_input_folder that contains text files with various formats. After the step below, you will see your merged pdf in the output folder!

python -m merge_files

5. Glimpses of output and different cases

5.1 Rerunning for new input folders

The script will automatically check whether or not an output folder has been created, and if so, skips to the next folder. This comes handy and saves time when you have add new input folders.

5.2 Progress bar for updates

The python library tqdm was used to add a progress bar while the text files are merged.

📖 Additional Notes

• If you have extremely large PDFs generated, the script will automatically split them to keep file sizes manageable.

• The default number of pages is set to 1000 for a part if the pdf has to be split. You can change this parameter in the _finalize_pdf function in conversion_utils.py script.

And you're all set!

For any questions, please contact s.narayanan@tilburguniversity.edu