Add mutual information calculation utility by vk3089790-arch · Pull Request #6769 · rizinorg/rizin

vk3089790-arch · 2026-09-19T17:15:37Z

Implements rz_mutual_info_init, rz_mutual_info_update, and rz_mutual_info_final in librz/util/mutual_info.c
Computes mutual information between two byte buffers using joint and marginal probability distributions
Includes unit tests covering perfectly correlated and independent data cases

Your checklist for this pull request

I've read the guidelines for contributing to this repository.
I made sure to follow the project's coding style.
I've documented every RZ_API function and struct this PR changes.
I've added tests that prove my changes are effective (required for changes to RZ_API).
I've updated the Rizin book with the relevant information (if needed).
I used Claude (Anthropic) to help me understand the mutual information
formula and to review my code for bugs (e.g. correct indexing when
computing marginal probabilities). I wrote the actual implementation
myself, filling in the core logic in rz_mutual_info_update and
rz_mutual_info_final line by line, and verified it compiles and passes
the included unit tests.

Detailed description

This PR implements a mutual information calculation utility for librz/util,
as requested in #5176.

What is mutual information?
Mutual information measures how much knowing the value of one variable
tells you about another — in this case, how correlated two byte buffers
are. It's useful for reverse engineering tasks like detecting related
fields in a file format, dependency between structure members, or
patterns in obfuscated data.

How it works
The implementation follows the same context-based pattern already used by
rz_entropy_* in librz/hash/algorithms/entropy/:

rz_mutual_info_init — initializes a context holding a 256x256 joint
frequency table.
rz_mutual_info_update — walks two equal-length byte buffers in
lockstep, incrementing the joint count for each observed byte pair.
Can be called multiple times to process data incrementally.
rz_mutual_info_final — derives marginal probabilities from the joint
table, then computes:

I(X;Y) = sum p(x,y) * log2(p(x,y) / (p(x) * p(y)))

over all non-zero joint probabilities.

Placement
This currently lives in librz/util/mutual_info.c as a general-purpose
utility, since it isn't tied to any specific tool. I'm following up with
maintainers on whether this should instead live alongside rz-diff or
rz-hash, and will move it if so.
...

Test plan
Ran locally:
ninja -C build
./build/test/unit/test_mutual_info

Both test_mutual_info_basic and test_mutual_info_independent pass.

...

Closing issues
Part of #5176

...

* Implements `rz_mutual_info_init`, `rz_mutual_info_update`, and `rz_mutual_info_final` in `librz/util/mutual_info.c` * Computes mutual information between two byte buffers using joint and marginal probability distributions * Includes unit tests covering perfectly correlated and independent data cases

wargio

handle when data is of 0 len

vk3089790-arch · 2026-09-25T01:13:32Z

@wargio Addressed all the review comments — added a zero-length guard in rz_mutual_info_final, and a check for zero marginals before the log2 division. All test cases pass locally. Re-requested your review, let me know if anything else is needed!

wargio

why the pic and the code is not used anywhere. also missing the spdx headers.

vk3089790-arch · 2026-09-26T13:00:21Z

@wargio Removed the stray profile photo commit that got included by accident, and added SPDX headers to all three files (mutual_info.c, rz_mutual_info.h, test_mutual_info.c). Pushed in the latest commits.

Rot127

The code is still unused. It needs to be put in action to be merged.

vk3089790-arch · 2026-09-26T15:26:01Z

Good point, understood — I'll wire this into a real command so it's actually used. Would you prefer this exposed via rz-diff (e.g. a mutual-information similarity mode) or rz-hash, or somewhere else? Happy to implement it once I know the preferred integration point.

wargio · 2026-09-26T17:11:30Z

i do not think this should go under rz-hash for any sort of reason.

This is purely a change needed in rz-diff

vk3089790-arch · 2026-09-27T02:13:53Z

Got it, thanks — I'll add this as a new mode/flag in rz-diff. Will look at how existing modes are structured there and follow the same pattern. Will push once it's working.

vk3089790-arch · 2026-09-27T07:16:30Z

Added a new -t mutualinfo mode to rz-diff that uses this API to compute mutual information between two files. Tested locally:

rz-diff -t mutualinfo README.md README.md → 5.268306 bits (identical)
rz-diff -t mutualinfo README.md CONTRIBUTING.md → 0.466583 bits (different files)

Pushed in the latest commit.

wargio

maybe im wrong but this is calculating the similarity, thus should go in the same code section as the other distances.

wargio

Also missing regression tests

…n tests Per review feedback, mutual information calculation now lives under the -d (distance) option family as 'mutualinfo', alongside myers, leven, lcs-roll, and ssdeep, rather than as a separate -t diff type. Since MI is measured in bits rather than a 0-1 similarity ratio, output uses a dedicated label instead of reusing 'similarity'. Added regression tests covering standard/JSON/quiet output modes.

vk3089790-arch · 2026-09-27T09:51:38Z

Moved mutual information into the -d distance path as -d mutualinfo (removed the old -t mutualinfo mode), and added regression tests in test/db/tools/rz_diff covering standard/JSON/quiet output. Since MI is reported in bits rather than a 0–1 similarity ratio like the other algorithms, I gave it its own output label ("bits" in JSON) instead of reusing similarity — happy to change that if you'd prefer a different format.

vk3089790-arch · 2026-09-27T16:41:47Z

@Rot127 This should also address your earlier comment — the API is now wired into rz-diff -d mutualinfo, so it's no longer unused. Let me know if you'd like anything else.

wargio · 2026-10-02T08:01:03Z

-		printf("similarity: %.3f\n", similarity);
-		if (ctx->distance != DIFF_DISTANCE_SSDEEP) {
+		if (ctx->distance == DIFF_DISTANCE_MUTUALINFO) {
+			printf("mutual information: %.6f bits\n", similarity);


same here 3 digits

Co-authored-by: Giovanni <561184+wargio@users.noreply.github.com>

wargio

i have checked the algo, i think we are missing some info.

we should report the bits, the coverage (i.e. 100.0 * min(lenA, lenB) / max(lenA, lenB) ) & overlap size (i.e. min(lenA, lenB))

Because the amount of info lost is kinda important, example:

lenA	lenB	coverage	overlap
800	1000	80%	800 bytes
800000	1000000	80%	800000 bytes

There is a clear difference of info lost between 200000 & 200

Per wargio's follow-up, mutual information distance now also reports coverage (percentage of the larger file's bytes covered by the comparison) and overlap (the actual number of bytes compared, i.e. min(a_size, b_size)), in standard, JSON, and quiet output modes. Also reduced bits precision from 6 to 3 decimals in standard/quiet output for readability, consistent with the other distance algorithms. Updated regression tests to match.

vk3089790-arch · 2026-10-02T09:24:55Z

@wargio I’ve fixed the issues you pointed out and pushed the changes to the add-mutual-info branch. I’ve also rebased the branch with the latest changes. Could you please take another look when you get a chance? Thanks!

vk3089790-arch · 2026-10-02T09:30:59Z

@wargio I’ve added the requested coverage and overlap information in 7cf99cc (rz-diff: add coverage and overlap to mutualinfo output). Coverage uses 100.0 * min(lenA, lenB) / max(lenA, lenB) and overlap uses min(lenA, lenB). I’ve requested another review. Thanks!

wargio

I can merge it once the last change is applied.

Co-authored-by: Giovanni <561184+wargio@users.noreply.github.com>

vk3089790-arch · 2026-10-04T10:10:04Z

@wargio Applied in 97bda8b — ready whenever you get a chance to take another look. Thanks for the quick feedback!

vk3089790-arch · 2026-10-05T08:54:33Z

@Rot127 This is now wired into rz-diff (librz/main/rz-diff.c), as discussed with @wargio. Could you take another look?

vk3089790-arch requested review from Rot127, b1llow, kazarmy, notxvilka, ret2libc, thestr4ng3r and wargio as code owners September 19, 2026 17:15

github-actions Bot added rz-test API RzUtil labels Sep 19, 2026

vk3089790-arch marked this pull request as draft September 19, 2026 17:22

vk3089790-arch marked this pull request as ready for review September 20, 2026 04:49

Add doc-comments to mutual information API

36d67f3

wargio requested changes Sep 23, 2026

View reviewed changes

Comment thread librz/util/mutual_info.c

Comment thread librz/util/mutual_info.c

Comment thread librz/util/mutual_info.c

vk3089790-arch added 2 commits September 24, 2026 21:27

Handle zero-length input in mutual information calculation

8df9406

Guard against zero marginals in mutual information sum

d5ecf90

vk3089790-arch requested a review from wargio September 25, 2026 01:11

wargio requested changes Sep 26, 2026

View reviewed changes

vk3089790-arch added 2 commits September 26, 2026 18:08

Add SPDX headers to mutual information files

67323df

Add SPDX headers to remaining mutual information files

1cc6156

vk3089790-arch force-pushed the add-mutual-info branch from 29a6372 to 1cc6156 Compare September 26, 2026 12:56

vk3089790-arch requested a review from wargio September 26, 2026 13:00

Rot127 requested changes Sep 26, 2026

View reviewed changes

vk3089790-arch requested a review from Rot127 September 26, 2026 15:26

Add mutualinfo type to rz-diff to use the mutual information API

a3a72c9

github-actions Bot added the rz-diff label Sep 27, 2026

wargio requested changes Sep 27, 2026

View reviewed changes

vk3089790-arch requested a review from wargio September 27, 2026 09:51

wargio reviewed Sep 27, 2026

View reviewed changes

Comment thread librz/main/rz-diff.c

Rot127 assigned notxvilka Sep 27, 2026

vk3089790-arch requested a review from wargio September 28, 2026 04:03

wargio reviewed Oct 2, 2026

View reviewed changes

Comment thread librz/main/rz-diff.c

wargio reviewed Oct 2, 2026

View reviewed changes

Update librz/main/rz-diff.c

f99641a

Co-authored-by: Giovanni <561184+wargio@users.noreply.github.com>

wargio requested changes Oct 2, 2026

View reviewed changes

vk3089790-arch requested a review from wargio October 2, 2026 09:25

wargio requested changes Oct 4, 2026

View reviewed changes

Comment thread librz/main/rz-diff.c Outdated

wargio reviewed Oct 4, 2026

View reviewed changes

Update librz/main/rz-diff.c

97bda8b

Co-authored-by: Giovanni <561184+wargio@users.noreply.github.com>

vk3089790-arch requested a review from wargio October 4, 2026 10:10

wargio approved these changes Oct 5, 2026

View reviewed changes

Uh oh!

Conversation

vk3089790-arch commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wargio left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vk3089790-arch commented Sep 25, 2026

Uh oh!

wargio left a comment

Choose a reason for hiding this comment

Uh oh!

vk3089790-arch commented Sep 26, 2026

Uh oh!

Rot127 left a comment

Choose a reason for hiding this comment

Uh oh!

vk3089790-arch commented Sep 26, 2026

Uh oh!

wargio commented Sep 26, 2026

Uh oh!

vk3089790-arch commented Sep 27, 2026

Uh oh!

vk3089790-arch commented Sep 27, 2026

Uh oh!

wargio left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wargio left a comment

Choose a reason for hiding this comment

Uh oh!

vk3089790-arch commented Sep 27, 2026

Uh oh!

vk3089790-arch commented Sep 27, 2026

Uh oh!

Uh oh!

Uh oh!

wargio Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

wargio left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vk3089790-arch commented Oct 2, 2026

Uh oh!

vk3089790-arch commented Oct 2, 2026

Uh oh!

Uh oh!

wargio left a comment

Choose a reason for hiding this comment

Uh oh!

vk3089790-arch commented Oct 4, 2026

Uh oh!

vk3089790-arch commented Oct 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

vk3089790-arch commented Sep 19, 2026 •

edited

Loading

wargio left a comment •

edited

Loading

wargio left a comment •

edited

Loading