fix: dedup json output by host+url when ip and port are empty

[ShubhamRasal]

Summary

-j (JSON output) only printed the first result for sources that don't return IP info (e.g. nerdydata).

Root cause

OutputWriter.WriteJsonData was deduping results using fmt.Sprintf("%s:%d", IP, Port). For sources where IP is empty and Port is 0, every result hashed to the same key :0, so all results after the first were dropped as duplicates.

Repro (before fix)

$ ./uncover -nerdydata jquery -silent -limit 10 -j
{"timestamp":...,"source":"nerdydata","ip":"","port":0,"host":"cloudflare.com","url":"https://www.cloudflare.com/"}
# (only one line)

Fix

Pick the dedup key based on what's populated:

• If IP or Port is set → key is IP:Port (preserves existing behaviour for shodan/censys/fofa, where two sources finding the same service should collapse into one row).
• Else → key is Host|Url (covers nerdydata and any future source that returns hosts/URLs only).

After fix

$ ./uncover -nerdydata jquery -silent -limit 10 -j
{...host:"cloudflare.com"...}
{...host:"wordpress.org"...}
{...host:"bootstrapcdn.com"...}
{...host:"jquery.com"...}
{...host:"amazonaws.com"...}
{...host:"mozilla.org"...}
{...host:"fontawesome.com"...}
{...host:"medium.com"...}
{...host:"creativecommons.org"...}
{...host:"bbc.co.uk"...}

Test plan

• ./uncover -nerdydata jquery -silent -limit 10 -j prints all 10 results
• ./uncover -nerdydata jquery -silent -limit 10 (non-JSON) still prints all 10 (unchanged code path)
• Verify shodan/censys IP:port dedup still collapses duplicates across sources

Summary by CodeRabbit

• Bug Fixes
  • Improved deduplication to avoid incorrectly skipping entries when port information is missing by using alternative identifiers (IP, host, URL) so unique records are preserved.

[neo-by-projectdiscovery-dev]

Neo - PR Security Review

No security issues found

_{Comment @pdneo help for available commands. · Open in Neo}

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 3216eb40-7c6e-4095-9c6f-842633ff6b90

📥 Commits

Reviewing files that changed from the base of the PR and between 177f89d and 4b14f4f.

📒 Files selected for processing (1)

• runner/output_writer.go

---

Walkthrough

WriteJsonData in runner/output_writer.go now computes a deduplication key conditionally: if data.IP is non-empty and data.Port is non-zero it uses IP:Port; otherwise it builds a key from Host and Url. The deduplication check and write path use this computed key.

Changes

Deduplication Key Fallback Logic

Layer / File(s)	Summary
Key Computation runner/output_writer.go	Compute dedup key conditionally: use IP:Port when data.IP non-empty and data.Port non-zero; otherwise build fallback key from data.Host and data.Url.
Dedup Check runner/output_writer.go	Use the computed key for the deduplication seen-check (early return if already seen).
Write Path runner/output_writer.go	Proceed to write JSON and mark key as seen using the same computed key (unchanged overall write flow).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Poem

A rabbit hops through code at night,
If IP's missing, Host and URL take flight.
Keys stitched gentle, duplicates stay thinned,
JSON stored once — the race is pinned. 🐇

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: fixing JSON deduplication to use host+url when IP and port are empty, which is the core issue addressed in the PR.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/json-output-dedup-empty-ip

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

runner/output_writer.go (1)

69-71: Use an unambiguous composite key for Host + Url.

data.Host + "|" + data.Url can theoretically collide if either field contains |. Consider escaping/quoting both parts.

Proposed hardening

-		key = data.Host + "|" + data.Url
+		key = fmt.Sprintf("%q|%q", data.Host, data.Url)

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@runner/output_writer.go` around lines 69 - 71, The composite key built as key
= data.Host + "|" + data.Url is ambiguous if Host or Url contain "|"—replace
this with an unambiguous encoding: encode or escape both data.Host and data.Url
and then join them, for example by JSON-serializing a two-element array or
base64-encoding each part before concatenation; update the place that assigns
key (the code using variable key and fields data.Host and data.Url) to use the
chosen safe encoding/escaping approach so the composite key cannot collide.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@runner/output_writer.go`:
- Around line 69-71: The composite key built as key = data.Host + "|" + data.Url
is ambiguous if Host or Url contain "|"—replace this with an unambiguous
encoding: encode or escape both data.Host and data.Url and then join them, for
example by JSON-serializing a two-element array or base64-encoding each part
before concatenation; update the place that assigns key (the code using variable
key and fields data.Host and data.Url) to use the chosen safe encoding/escaping
approach so the composite key cannot collide.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8d2fffd8-fda9-434f-9abc-d1a3d967d5cf

📥 Commits

Reviewing files that changed from the base of the PR and between 92e6147 and 177f89d.

📒 Files selected for processing (1)

• runner/output_writer.go