Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,8 @@ program. Several classic scripts given in one run share one global scope, as scr
do: a top-level `var`, function, `let`, `const` or `class` of one is treated as read and written by
the others when another input names it (a free reference or a top-level declaration of its own) or
reaches globals through a computed key or code text, and is then not inlined away, removed or
renamed. A built-in name (`eval`, `String`, `atob`, ...) that another input declares at its top level, assigns as a bare name, sets as a property of the global object (`this.X =`, `globalThis.X =`, `window["X"] =`, a definer such as `Object.defineProperty(globalThis, ...)`), or declares or assigns in a code text it runs (`eval`, `(0, eval)`, `Function`, a timer string) is not the built-in for any input, so no fold relies on it. A global-object write with a computed key, or a code text that is not a literal, counts for every name. In the default lane an input whose own built-in scan is not clean (it uses a built-in, the global object or a code runner outside the listed harmless shapes, or patches a built-in member) counts for every name too. The default lane also assumes no input hands code text to a DOM API that evaluates it (`document.write`, a `javascript:` URL, a script element's text, an event-handler attribute string). Renaming never renames a classic script's top-level bindings in the safe lane. Under `--unsafe`,
renamed, in both lanes; a string array, decoder, decoder alias or protection named that way is kept
and nothing is decoded through it. A built-in name (`eval`, `String`, `atob`, ...) that another input declares at its top level, assigns as a bare name, sets as a property of the global object (`this.X =`, `globalThis.X =`, `window["X"] =`, a definer such as `Object.defineProperty(globalThis, ...)`), or declares or assigns in a code text it runs (`eval`, `(0, eval)`, `Function`, a timer string) is not the built-in for any input, so no fold relies on it. A global-object write with a computed key, or a code text that is not a literal, counts for every name. In the default lane an input whose own built-in scan is not clean (it uses a built-in, the global object or a code runner outside the listed harmless shapes, or patches a built-in member) counts for every name too. The default lane also assumes no input hands code text to a DOM API that evaluates it (`document.write`, a `javascript:` URL, a script element's text, an event-handler attribute string). Renaming never renames a classic script's top-level bindings in the safe lane. Under `--unsafe`,
when two or more classic scripts are given or another input may observe the global object's
names, they keep their names too (`--rename unique`, whose counter is shared by every input,
still renames them when only other scripts are the reason), and no input's new top-level name is a
Expand Down Expand Up @@ -262,7 +263,9 @@ are decoded, even when the program enumerates or tests the global object's names
them by key, through `Object.assign`, `Reflect.set`, `Object.defineProperty` or a loop over keys,
or runs code text through any runner, `(0, eval)`,
`Function`, a timer string, or a runner that code text hands back such as
`Function("return eval")()(s)`). A few passes that rest on
`Function("return eval")()(s)`). No function declared in a block overwrites a decoder alias, a
decoder wrapper or a constant used as a decoder argument through Annex B hoisting (`var d = dec;
{ function d() {} } d(1)` is decoded as a call of `dec`). A few passes that rest on
heuristics are allowed to run. These are `template-literals` and JSX reconstruction (`jsx`, `jsx-new`).
`dead-code`, which deletes branches behind constant comparisons, runs in both lanes.
`fold-pure-calls` also folds calls to a pure helper that is passed around as a value or that the
Expand Down
14 changes: 10 additions & 4 deletions crates/untwist-deobfuscate/src/bindings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ use oxc_syntax::symbol::{SymbolFlags, SymbolId};
use rustc_hash::FxHashSet;
use untwist_core::pass::Cx;
use untwist_core::semantic::{block_function_symbols, declared_in_block, AnnexBWrites};
use untwist_core::semantic::{observers, Observers};
use untwist_core::semantic::{is_shared_global, observers, Observers};

pub fn resolved_symbol(reference: &IdentifierReference<'_>, scoping: &Scoping) -> Option<SymbolId> {
scoping
Expand Down Expand Up @@ -112,6 +112,9 @@ pub fn reaches_a_script_global(
let Some((names, props)) = script_global_names(symbols, program, scoping) else {
return false;
};
if names.iter().copied().any(is_shared_global) {
return true;
}
untwist_core::semantic::observing_open(|| {
names_reached_beside(program, scoping, (&names, &props), removed)
})
Expand All @@ -123,12 +126,15 @@ pub fn reaches_a_script_global_once_removed<'a>(
program: &Program<'a>,
cx: &Cx<'a, '_>,
) -> bool {
if cx.opts.unsafe_transforms || observers() == Observers::None {
return false;
}
let Some((names, props)) = script_global_names(symbols, program, cx.scoping()) else {
return false;
};
if names.iter().copied().any(is_shared_global) {
return true;
}
if cx.opts.unsafe_transforms || observers() == Observers::None {
return false;
}
let mut pruned = program.clone_in(cx.allocator());
Prune {
removed: removed.iter().copied().collect(),
Expand Down
14 changes: 14 additions & 0 deletions crates/untwist-deobfuscate/src/calls/agnostic/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,20 @@ pub fn names_reached_quietly<'a>(
names_reach(program, scoping, watch).reach
}

pub fn names_observed<'a>(
program: &'a Program<'a>,
scoping: &Scoping,
names: &[&str],
) -> Option<Span> {
let watch = ReachWatch {
names,
props: names,
fns: &[],
observe: true,
};
names_reach(program, scoping, watch).observed
}

#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct NamesReached {
pub reached: bool,
Expand Down
1 change: 0 additions & 1 deletion crates/untwist-deobfuscate/src/calls/agnostic/observe.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,6 @@ impl<'a> Sites<'_, 'a> {
self.watch.observe
&& !self.watch.props.is_empty()
&& self.out.observed.is_none()
&& self.out.reach.is_none()
&& !self.in_machinery(span)
}

Expand Down
7 changes: 7 additions & 0 deletions crates/untwist-deobfuscate/src/decoded.rs
Original file line number Diff line number Diff line change
Expand Up @@ -759,6 +759,13 @@ fn judge_runs<'p>(
let unit = run.request.unit;
let deferred = root_array_name(scoping, &run).filter(|_| watched && !spelled);
let reached = reached.as_ref().filter(|_| !spelled);
if !nested_root && reach::names_shared(program, scoping, &run) {
let span = run.request.program.string_array.function.span;
decoded
.notes
.push((notes::STRING_ARRAY_GLOBAL_REACH.to_string(), Some(span)));
continue;
}
let refusal = refused_before_judging(&run, reached, &decoding);
let speculated = refusal
.filter(|(note, _)| watched && *note == notes::STRING_ARRAY_GLOBAL_REACH)
Expand Down
30 changes: 30 additions & 0 deletions crates/untwist-deobfuscate/src/decoded/reach.rs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ use crate::bridge::DecodePlan;
use crate::calls::agnostic::{global_reach, ReachWatch};
use crate::mutants::decoder_name_watch_dropped;
use crate::root_reach::root_alias_names;
use crate::stamp::original_name;
use untwist_core::semantic::is_shared_global;

pub(super) type Reached = (FxHashSet<usize>, Span, Option<&'static str>);

Expand Down Expand Up @@ -58,6 +60,34 @@ fn watched_units<'p>(
watched
}

pub(super) fn names_shared(
program: &Program<'_>,
scoping: &Scoping,
run: &UnitRun<'_, '_>,
) -> bool {
let root = scoping.root_scope_id();
let roots: Vec<(SymbolId, &str)> = run
.request
.guard
.symbols
.iter()
.filter(|(symbol, _)| scoping.symbol_scope_id(*symbol) == root)
.copied()
.collect();
let targets: FxHashSet<SymbolId> = roots.iter().map(|(symbol, _)| *symbol).collect();
roots
.iter()
.flat_map(|(symbol, name)| {
[
Some(*name),
original_name(program.source_text, scoping, *symbol),
]
})
.flatten()
.chain(root_alias_names(program, scoping, &targets))
.any(is_shared_global)
}

pub(super) fn watched<'p>(
program: &'p Program<'_>,
scoping: &Scoping,
Expand Down
5 changes: 5 additions & 0 deletions crates/untwist-deobfuscate/src/init_order/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -524,6 +524,11 @@ pub(crate) fn early_or_written(
if early.contains(&symbol) {
return Some(notes::MACHINERY_READ_BEFORE_DECLARATION);
}
if scoping.symbol_scope_id(symbol) == scoping.root_scope_id()
&& untwist_core::semantic::is_shared_global(scoping.symbol_name(symbol))
{
return Some(notes::MACHINERY_SHARED_GLOBAL);
}
(!early_read_guards_dropped() && constant_written(scoping, symbol))
.then_some(notes::MACHINERY_CONSTANT_WRITTEN)
}
Expand Down
65 changes: 61 additions & 4 deletions crates/untwist-deobfuscate/src/legacy_reach.rs
Original file line number Diff line number Diff line change
Expand Up @@ -102,12 +102,22 @@ pub(crate) fn widen_layered(token: &str, used: Vec<String>, points_to: bool) {
}

pub(crate) fn drop_layered(token: &str) {
verify(token);
DEFERRAL.with(|deferral| {
deferral
.borrow_mut()
let mut deferral = deferral.borrow_mut();
if let Some(last) = deferral
.pending
.iter()
.rposition(|pending| pending == token)
{
deferral.pending.remove(last);
}
if let Some(last) = deferral
.layered
.retain(|(pending, _)| pending != token)
.iter()
.rposition(|(pending, _)| pending == token)
{
deferral.layered.remove(last);
}
});
}

Expand Down Expand Up @@ -151,6 +161,20 @@ pub(crate) fn waive_observation(names: &[&str]) -> bool {
})
}

pub(crate) fn observe_at_stage_end(names: &[&str]) -> bool {
DEFERRAL.with(|deferral| {
let mut deferral = deferral.borrow_mut();
if !deferral.open || names.is_empty() {
return false;
}
let names: Vec<String> = names.iter().map(|name| (*name).to_string()).collect();
if !deferral.observations.contains(&names) {
deferral.observations.push(names);
}
true
})
}

pub fn settle_observations(mut observed: impl FnMut(&[&str]) -> bool) {
let observations =
DEFERRAL.with(|deferral| std::mem::take(&mut deferral.borrow_mut().observations));
Expand Down Expand Up @@ -230,3 +254,36 @@ impl Drop for Rerunnable {
DEFERRAL.with(|deferral| deferral.borrow_mut().rerunnable = self.0);
}
}

#[cfg(test)]
mod tests {
use super::*;

fn held(name: &str) -> Layered {
Layered {
names: vec![name.to_string()],
..Layered::default()
}
}

fn counts() -> (usize, usize) {
DEFERRAL.with(|deferral| {
let deferral = deferral.borrow();
(deferral.pending.len(), deferral.layered.len())
})
}

#[test]
fn dropping_a_later_speculation_keeps_the_obligation_an_earlier_round_kept() {
forget();
defer_layered("layered:a".to_string(), held("a"));
defer_layered("layered:a".to_string(), held("a"));
drop_layered("layered:a");
let kept = counts();
drop_layered("layered:a");
let dropped = counts();
forget();
assert_eq!(kept, (1, 1));
assert_eq!(dropped, (0, 0));
}
}
4 changes: 4 additions & 0 deletions crates/untwist-deobfuscate/src/notes.rs
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,9 @@ pub const MACHINERY_READ_BEFORE_DECLARATION: &str =
pub const MACHINERY_CONSTANT_WRITTEN: &str =
"the string array or a decoder is a const that code assigns to, and that assignment throws, so \
it was left unstamped and its call sites undecoded";
pub const MACHINERY_SHARED_GLOBAL: &str =
"another input given in the same run names the string array or a decoder, and the inputs \
share one global scope, so it was left unstamped and its call sites undecoded";
pub const LEGACY_ARRAY_REACHED: &str =
"this plain string array may be reached by name through the global object, a with statement \
or a direct eval, so it was not turned into a string-array function and was left undecoded";
Expand Down Expand Up @@ -667,6 +670,7 @@ catalogue!(
STRING_ARRAY_DECLARED_UNDEFINED,
MACHINERY_READ_BEFORE_DECLARATION,
MACHINERY_CONSTANT_WRITTEN,
MACHINERY_SHARED_GLOBAL,
LEGACY_ARRAY_REACHED,
LEGACY_DECODER_BEFORE_ARRAY,
LEGACY_ROTATOR_AFTER_CODE,
Expand Down
44 changes: 37 additions & 7 deletions crates/untwist-deobfuscate/src/removal.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ use rustc_hash::{FxHashMap, FxHashSet};
use untwist_core::ast_util::edit::StmtEdits;
use untwist_core::ast_util::matcher::var_function_or_declaration;
use untwist_core::pass::{Cx, Interest, Pass};
use untwist_core::semantic::{observers, Observers};
use untwist_core::semantic::{is_shared_global, observers, Observers};

use oxc_semantic::Scoping;
use oxc_syntax::symbol::SymbolId;
Expand Down Expand Up @@ -49,10 +49,16 @@ fn observed_names_of(
nested_root: bool,
(stamped, units): (&Stamped<'_>, &[Machinery]),
) -> Observation {
if nested_root
|| untwist_core::semantic::has_module_syntax(program)
|| observers() == Observers::None
{
if nested_root || untwist_core::semantic::has_module_syntax(program) {
return Observation::default();
}
if let Some(span) = shared_machinery((program.source_text, scoping), (stamped, units)) {
return Observation {
observed: Some(span),
outside_premise: None,
};
}
if observers() == Observers::None {
return Observation::default();
}
let root = scoping.root_scope_id();
Expand Down Expand Up @@ -86,11 +92,35 @@ fn observed_names_of(
#[cfg(feature = "refusal-spans")]
record_observed(&reach);
Observation {
observed: reach.observed,
observed: reach.observed.or(reach.reach),
outside_premise: reach.outside_premise,
}
}

fn shared_machinery(
(source_text, scoping): (&str, &Scoping),
(stamped, units): (&Stamped<'_>, &[Machinery]),
) -> Option<Span> {
let root = scoping.root_scope_id();
let shared_fn = stamped
.arrays
.iter()
.chain(stamped.decoders.iter())
.filter_map(|entry| Some((entry, entry.symbol?)))
.filter(|(_, symbol)| scoping.symbol_scope_id(*symbol) == root)
.find(|(entry, symbol)| {
is_shared_global(entry.name)
|| original_name(source_text, scoping, *symbol).is_some_and(is_shared_global)
})
.map(|(entry, _)| entry.statement);
shared_fn.or_else(|| {
units
.iter()
.find(|unit| unit.alias_names().any(is_shared_global))
.and_then(|unit| unit.statements().first().copied())
})
}

#[cfg(feature = "refusal-spans")]
fn record_observed(reach: &crate::calls::agnostic::GlobalReach) {
let Some(span) = reach.observed else {
Expand Down Expand Up @@ -330,7 +360,7 @@ fn dead_root_aliases<'p>(
read: FxHashSet::default(),
};
read.visit_program(program);
candidates.retain(|alias| !read.read.contains(alias.name));
candidates.retain(|alias| !read.read.contains(alias.name) && !is_shared_global(alias.name));
candidates
}

Expand Down
24 changes: 23 additions & 1 deletion crates/untwist-deobfuscate/src/root_reach.rs
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ use rustc_hash::{FxHashMap, FxHashSet};
use untwist_core::pass::Cx;
use untwist_core::semantic::has_module_syntax;

use crate::calls::agnostic::names_reached_quietly;
use crate::calls::agnostic::{names_observed, names_reached_quietly};
use crate::decoded::syntactic_global_reach;
use crate::legacy_reach;
use crate::mutants::root_alias_reach_trusted;
Expand All @@ -33,6 +33,28 @@ pub(crate) fn root_names_reached(
names_reached_quietly(program, scoping, names)
}

pub(crate) fn observed_roots(
program: &Program<'_>,
cx: &Cx<'_, '_>,
bindings: &[SymbolId],
) -> Option<FxHashSet<SymbolId>> {
let script = !cx.opts.nested_root && !has_module_syntax(program);
if cx.opts.unsafe_transforms || !script || root_alias_reach_trusted() {
return None;
}
let scoping = cx.scoping();
let bindings = root_symbols(scoping, bindings);
let names = names_of(program.source_text, scoping, &bindings);
if names.is_empty() {
return None;
}
names_observed(program, scoping, &names)?;
if legacy_reach::observe_at_stage_end(&names) {
return None;
}
Some(bindings.into_iter().collect())
}

pub(crate) struct ReachedRoots {
pub bindings: FxHashSet<SymbolId>,
pub targets: FxHashSet<SymbolId>,
Expand Down
5 changes: 5 additions & 0 deletions crates/untwist-deobfuscate/src/string_array/legacy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ use untwist_core::ast_util::respan::SYNTHETIC_BASE;
use untwist_core::ast_util::search::statements_at;
use untwist_core::parse::{has_error, parse_with, SourceType};
use untwist_core::pass::Cx;
use untwist_core::semantic::is_shared_global;

use super::array_declaration;
use super::checksum::{
Expand Down Expand Up @@ -705,6 +706,10 @@ fn refusal(
if let Some(note) = dynamic.refusal(scoping, legacy.symbol) {
return Some((note.to_string(), statement));
}
let root = scoping.symbol_scope_id(legacy.symbol) == scoping.root_scope_id();
if root && is_shared_global(scoping.symbol_name(legacy.symbol)) {
return Some((notes::LEGACY_ARRAY_REACHED.to_string(), statement));
}
if let Some(span) = reached_by_name(program, cx, legacy) {
if cx.opts.unsafe_transforms || !legacy_reach::defers() {
return Some((notes::LEGACY_ARRAY_REACHED.to_string(), span));
Expand Down
Loading
Loading