async impl with tokio & improvement on hotpaths and webdav improvements with reverse proxies

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "irondrop"
-version = "2.7.0"
+version = "2.7.2"
 edition = "2024"
 license = "MIT"
 description = "Drop files, not dependencies - a well tested fully featured & battle-ready server in a single Rust binary with support for indexing through 10M files."
@@ -15,13 +15,14 @@ env_logger = "0.11.10"
 base64 = "0.22.1"
 rustls = { version = "0.23", default-features = false, features = ["ring", "std", "logging", "tls12"] }
 rustls-pemfile = "2"
+tokio = { version = "1.47", features = ["fs", "io-util", "macros", "net", "rt-multi-thread", "sync", "time"] }
+tokio-rustls = "0.26"
 
 
 [dev-dependencies]
 reqwest = { version = "0.13", features = ["blocking", "json"] }
 serde_json = "1.0"
 tempfile = "3.27"
-threadpool = "1.8.1"
 rcgen = "0.14"
 
 [profile.release]
@@ -38,5 +39,3 @@ overflow-checks = false   # Disable integer overflow checks
 [profile.dist]
 inherits = "release"
 lto = "thin"
-
-

README.md

@@ -25,8 +25,8 @@ IronDrop focuses on predictable behavior, simplicity, and low overhead. Use it t
 - Basic security features: rate limiting, optional Basic Auth, path safety checks
 - Native SSL/TLS support via `--ssl-cert` and `--ssl-key` (built-in HTTPS, no reverse proxy required)
 - Single binary; templates and assets are embedded
-- Core engine is dependency-free in critical paths: networking, search, and filesystem access are implemented in-house
-- Standard production dependencies are still used where practical (for example `clap`, `log`/`env_logger`, and `rustls`)
+- Core HTTP layer is implemented in-house (request parsing, routing, streaming) without an external HTTP framework
+- Tokio is used for the async runtime and networking; standard production dependencies are used where practical (for example `clap`, `log`/`env_logger`, `tokio`, and `rustls`/`tokio-rustls`)
 - Ultra-compact search index option for very large directory trees (tested up to ~10M entries)
 - WebDAV (RFC 4918 Class 1 + Class 2 core): `OPTIONS`, `PROPFIND`, `PROPPATCH`, `MKCOL`, `PUT`, `DELETE`, `COPY`, `MOVE`, `LOCK`, `UNLOCK`
   - Enabled only when `--enable-webdav true` (or equivalent config setting) is provided
@@ -38,7 +38,7 @@ IronDrop includes an RFC 4918-focused implementation. The WebDAV core engine is
 - Supported methods: `OPTIONS`, `PROPFIND`, `PROPPATCH`, `MKCOL`, `PUT`, `DELETE`, `COPY`, `MOVE`, `LOCK`, `UNLOCK`
 - WebDAV is feature-gated and disabled by default; enable explicitly with `--enable-webdav true`
 - Capability headers: `DAV: 1,2`, `Allow`, `MS-Author-Via`
-- `PROPFIND`: `allprop`, `propname`, named `prop`, per-property `propstat` grouping (`200`/`404`), and finite-depth refusal (`403` + `propfind-finite-depth`)
+- `PROPFIND`: `allprop`, `propname`, named `prop`, per-property `propstat` grouping (`200`/`404`), with `Depth: 0`, `1`, and recursive `infinity`
 - `PROPPATCH`: dead-property `set`/`remove` with `207 Multi-Status` results
 - Locking: exclusive write locks, lock refresh, `If` header token evaluation (including `Not` conditions), and token-gated write preconditions
 - Tree operations: lock-aware `DELETE` multistatus behavior (`207` with `423`/`424` where applicable)
@@ -53,7 +53,7 @@ Current RFC scope limits:
 Designed to keep memory usage steady and to stream large files without buffering them in memory. The ultra-compact search mode reduces memory for very large directory trees.
 
 - Ultra-compact search: approximately ~110 MB of RAM for around 10 million paths; search latency depends on CPU, disk, and query specifics.
-- Dependency profile: networking/search/filesystem core paths are dependency-free, while operational dependencies such as `clap`, `log`/`env_logger`, and `rustls` are used as stable standard building blocks.
+- Dependency profile: the HTTP implementation is in-house; Tokio provides async scheduling and networking, and dependencies such as `clap`, `log`/`env_logger`, and `rustls`/`tokio-rustls` are used as stable standard building blocks.
 
 ## Security
 
@@ -63,42 +63,16 @@ Includes native SSL/TLS (HTTPS), rate limiting, optional Basic Auth, basic input
 
 Getting started with IronDrop is simple.
 
-### From Source
+### From crates.io
 
 ```bash
-# Clone the repository
-git clone https://github.com/dev-harsh1998/IronDrop.git
-cd IronDrop
-
-# Build the release binary
-cargo build --release
-
-# The executable will be in ./target/release/irondrop
+cargo install irondrop
 ```
 
-### System-Wide Installation (Recommended)
-
-To use IronDrop from anywhere on your system, install it to a directory in your PATH:
+### Latest from Git
 
 ```bash
-# Linux/macOS - Install to /usr/local/bin (requires sudo)
-sudo cp ./target/release/irondrop /usr/local/bin/
-
-# Alternative: Install to ~/.local/bin (no sudo required)
-mkdir -p ~/.local/bin
-cp ./target/release/irondrop ~/.local/bin/
-# Add ~/.local/bin to PATH if not already:
-echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc  # or ~/.zshrc
-source ~/.bashrc  # or restart terminal
-
-# Windows (PowerShell as Administrator)
-# Create program directory
-New-Item -ItemType Directory -Force -Path "C:\Program Files\IronDrop"
-# Copy executable
-Copy-Item ".\target\release\irondrop.exe" "C:\Program Files\IronDrop\"
-# Add to system PATH (requires restart or new terminal)
-$env:PATH += ";C:\Program Files\IronDrop"
-[Environment]::SetEnvironmentVariable("PATH", $env:PATH, [EnvironmentVariableTarget]::Machine)
+cargo install --git https://github.com/dev-harsh1998/IronDrop.git
 ```
 
 **Verify Installation:**
@@ -114,21 +88,18 @@ irondrop -d ~/Documents --listen 0.0.0.0
 
 ### Quick start
 
-**Step 1:** Download or build IronDrop
+**Step 1:** Install IronDrop
 ```bash
-# Build from source (requires Rust)
-git clone https://github.com/dev-harsh1998/IronDrop.git
-cd IronDrop
-cargo build --release
+cargo install irondrop
 ```
 
 **Step 2:** Start sharing files immediately
 ```bash
 # Share your current directory (safest - local access only)
-./target/release/irondrop -d .
+irondrop -d .
 
 # Share with your network (accessible to other devices)
-./target/release/irondrop -d . --listen 0.0.0.0
+irondrop -d . --listen 0.0.0.0
 ```
 
 **Step 3:** Open your browser and visit `http://localhost:8080`
@@ -194,16 +165,25 @@ location / {
 ```
 
 **Subpath Configuration (e.g., `/webstorage/`):**
+
+Start IronDrop with the `--base-path` flag:
+```bash
+irondrop -d ./files --base-path /webstorage --listen 0.0.0.0
+```
+
+Then configure Nginx to forward the full path (no stripping needed):
 ```nginx
 location /webstorage/ {
-    proxy_pass http://127.0.0.1:8080/;
-    proxy_redirect / /webstorage/;
-    sub_filter 'href="/"' 'href="/webstorage/"';
-    sub_filter_once off;
-    # ... see deployment guide for full sub_filter list
+    proxy_pass http://127.0.0.1:8080/webstorage/;
+    proxy_set_header Host $host;
+    proxy_set_header X-Real-IP $remote_addr;
+    client_max_body_size 0;
+    proxy_buffering off;
 }
 ```
 
+> **Note:** No `sub_filter` hacks or extra `/_irondrop/` location blocks are needed. The `--base-path` flag makes IronDrop fully reverse-proxy aware — all generated URLs (HTML links, JavaScript API calls, WebDAV hrefs, HTTP redirects) are automatically prefixed with the configured base path.
+
 See the [Deployment Guide](./doc/DEPLOYMENT.md#nginx-reverse-proxy-deployment) for full configuration examples and optimization settings.
 
 ### 🛠️ Configuration Options
@@ -212,15 +192,17 @@ See the [Deployment Guide](./doc/DEPLOYMENT.md#nginx-reverse-proxy-deployment) f
 IronDrop offers extensive customization through command-line arguments:
 
 | Option | Description | Example |
-|--------|-------------|----------|
+|--------|-------------|---------|
 | `-d, --directory` | **Required** - Directory to serve | `-d /home/user/files` |
 | `-l, --listen` | Listen address (default: 127.0.0.1) | `-l 0.0.0.0` |
 | `-p, --port` | Port number (default: 8080) | `-p 3000` |
 | `--enable-upload` | Enable file uploads | `--enable-upload true` |
 | `--enable-webdav` | Enable WebDAV methods (`OPTIONS`, `PROPFIND`, `PROPPATCH`, `MKCOL`, `PUT`, `DELETE`, `COPY`, `MOVE`, `LOCK`, `UNLOCK`) | `--enable-webdav true` |
+| `--disable-rate-limit` | Disable rate limiting **only when WebDAV is enabled** | `--enable-webdav true --disable-rate-limit true` |
+| `--base-path` | Base URL path prefix for reverse proxy sub-path deployments | `--base-path /webstorage` |
 | `--username/--password` | Basic authentication | `--username admin --password secret` |
 | `-a, --allowed-extensions` | Restrict file types | `-a "*.pdf,*.doc,*.zip"` |
-| `-t, --threads` | Worker threads (default: 8) | `-t 16` |
+| `-t, --threads` | Tokio runtime worker threads (default: 8) | `-t 16` |
 | `--config-file` | Use INI configuration file | `--config-file prod.ini` |
 | `--ssl-cert` | SSL certificate file (PEM) for HTTPS | `--ssl-cert cert.pem` |
 | `--ssl-key` | SSL private key file (PEM) for HTTPS | `--ssl-key key.pem` |
@@ -245,8 +227,11 @@ WebDAV can also be enabled in config:
 ```ini
 [webdav]
 enable_webdav = true
+disable_rate_limit = false
 ```
 
+Note: `disable_rate_limit` is ignored unless WebDAV is enabled.
+
 Quick CLI example:
 
 ```bash
@@ -293,7 +278,9 @@ For comprehensive documentation, see our [Complete Documentation Index](./doc/RE
 
 ## Version notes
 
-Recent releases include direct-to-disk uploads, an ultra-compact search mode, and a `/monitor` page with a JSON endpoint.
+Recent releases include:
+- **v2.7.2**: Major memory optimizations: WebDAV `Transfer-Encoding: chunked` background thread streaming (near-zero RAM usage for large tree traversals), Web UI 1,000-file pagination with lazy metadata loading, and a massive ~340MB reduction in default search index bootstrap footprint.
+- **v2.7.1**: Direct-to-disk uploads, ultra-compact search mode, and a `/monitor` page with a JSON endpoint.
 
 ## Documentation
 
@@ -321,10 +308,10 @@ IronDrop has extensive documentation covering its architecture, API, and feature
 
 ## Testing
 
-IronDrop is rigorously tested with **272 automated tests**:
+IronDrop is rigorously tested with **325 automated tests**:
 
-- **48 unit tests** in core source modules
-- **224 integration/system tests** across **28** test files (including WebDAV RFC suites)
+- **44 unit tests** in core source modules
+- **281 integration/system tests** across **30** test files (including WebDAV RFC suites)
 
 ### Coverage Areas
 - HTTP parser/request handling, auth, rate limiting, monitoring, uploads, search, and utilities
@@ -356,7 +343,7 @@ IronDrop is licensed under the [MIT License](./LICENSE).
 <div align="center">
   <p>
     <strong>Made with ❤️ and 🦀 in Rust</strong><br>
-    <em>Dependency-free core engine paths • Production ready • Battle tested with 272 automated tests</em>
+    <em>Dependency-free core engine paths • Production ready • Battle tested with 325 automated tests</em>
   </p>
   <p>
     <a href="https://github.com/dev-harsh1998/IronDrop">⭐ Star us on GitHub</a>

config/irondrop.ini

@@ -76,6 +76,13 @@ chunk_size = 2048
 # • For uploads, IronDrop needs write permissions too
 directory = .
 
+# 🛤️ Base Path - Run IronDrop under a URL sub-path (for reverse proxies)
+# • Default: empty (serves from root /)
+# • Useful when running behind Nginx: e.g., location /webstorage/
+# • Example: /webstorage
+# • Prevents the need for Nginx sub_filter URL rewriting hacks
+# base_path = /webstorage
+
 # ===============================================================================
 # ⬆️  UPLOAD SYSTEM CONFIGURATION
 # ===============================================================================
@@ -126,6 +133,13 @@ max_upload_size = 5GB
 # ⚠️  Recommendation: Use authentication and HTTPS when exposing WebDAV externally.
 enable_webdav = false
 
+# 🧯 Disable WebDAV Rate Limiting (WebDAV-only)
+# • false = Keep rate limiting enabled (recommended default)
+# • true  = Disable rate limiting, but ONLY when WebDAV is enabled
+# • Ignored automatically if enable_webdav = false
+# • Useful for Finder / DAV clients that burst metadata requests
+disable_rate_limit = false
+
 
 # ===============================================================================
 # 🔒 SECURITY CONFIGURATION
@@ -275,6 +289,9 @@ detailed = true
 # cert = /etc/letsencrypt/live/files.example.com/fullchain.pem
 # key = /etc/letsencrypt/live/files.example.com/privkey.pem
 # 
+# # Example of running under a sub-path proxy:
+# # base_path = /company-files
+# 
 # [upload]
 # enable_upload = true
 # max_upload_size = 100MB
@@ -322,6 +339,7 @@ detailed = true
 # ✅ 7. Add authentication for network access (set username/password)
 # ✅ 8. Configure allowed file extensions for security
 # ✅ 8b. (Optional) Add SSL cert and key for HTTPS
+# ✅ 8c. (Optional) Set base_path if deploying behind a proxy sub-path
 # ✅ 9. Run: irondrop --config-file my-config.ini
 # ✅ 10. Open browser: http://localhost:8080 (or your chosen port)
 # ✅ 11. Enjoy blazing-fast file sharing! 🚀