Skip to content

Commit d97aeee

Browse files
fix(llm): fall back to sampling defaults for options passed as undefined (#1611)
## Description `sample()` merged the caller's config over the defaults with an object spread, so an option passed as `undefined` (for example `{ temperature: settings.temperature }` where the setting is unset) replaced its default with `undefined`. The native sampler reads these options as required numbers and throws `INVALID_ARGUMENT`, which fails `generate` and chat sessions on every LLM. This change resolves each default with `??`, for the top-level options and for the `dry` and `xtc` sub-configs, so an `undefined` field behaves like an omitted one. A fake-runtime test covers the case. ### Introduces a breaking change? - [ ] Yes - [x] No ### Type of change - [x] Bug fix (change which fixes an issue) - [ ] New feature (change which adds functionality) - [ ] Documentation update (improves or adds clarity to existing documentation) - [ ] Other (chores, tests, code style improvements etc.) ### Tested on - [ ] iOS - [ ] Android TypeScript-only change, verified with the fake-runtime API tests. Not run on a device. ### Testing instructions 1. `yarn workspace react-native-executorch test __tests__/extensions/sampler.test.ts` 2. The new test `falls back to the defaults for options passed as undefined` passes. Without the change to `sampler.ts`, it fails because `temperature`, `topK`, `topP`, `minP`, `repetitionPenalty`, `penaltyLastN`, and the `dry`/`xtc` defaults reach the native call as `undefined`. ### Screenshots Not applicable. ### Related issues Fixes #1610 ### Checklist - [x] I have performed a self-review of my code - [x] I have commented my code, particularly in hard-to-understand areas - [ ] I have updated the documentation accordingly - [x] My changes generate no new warnings ### Additional notes The native side reads these options with `getRequiredProperty` in `cpp/extensions/llm/sampler.cpp`, which accepts a missing key only through the TypeScript defaults. Documentation needs no update, since the documented defaults are unchanged.
1 parent d35ad55 commit d97aeee

2 files changed

Lines changed: 56 additions & 3 deletions

File tree

‎packages/react-native-executorch/__tests__/extensions/sampler.test.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,4 +67,43 @@ describe('sample — options merging', () => {
6767
logits.dispose();
6868
}
6969
});
70+
71+
it('falls back to the defaults for options passed as undefined', () => {
72+
const logits = tensor('float32', [1, 10]);
73+
try {
74+
sample(
75+
logits,
76+
{ generatedTokens: new Int32Array(0) },
77+
{
78+
temperature: undefined,
79+
topK: undefined,
80+
topP: undefined,
81+
minP: undefined,
82+
repetitionPenalty: undefined,
83+
penaltyLastN: undefined,
84+
dry: { multiplier: 0.8, base: undefined, allowedLength: undefined },
85+
xtc: { probability: 0.5, threshold: undefined },
86+
}
87+
);
88+
89+
expect(llm.lastSampleOptions()).toMatchObject({
90+
temperature: 0.8,
91+
repetitionPenalty: 1.0,
92+
penaltyLastN: -1,
93+
topK: 40,
94+
topP: 0.95,
95+
minP: 0.05,
96+
dry: {
97+
multiplier: 0.8,
98+
base: 1.75,
99+
allowedLength: 2,
100+
penaltyLastN: -1,
101+
sequenceBreakers: new Int32Array(0),
102+
},
103+
xtc: { probability: 0.5, threshold: 0.1 },
104+
});
105+
} finally {
106+
logits.dispose();
107+
}
108+
});
70109
});

‎packages/react-native-executorch/src/extensions/llm/sampler.ts‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -171,11 +171,25 @@ const DEFAULT_XTC_OPTIONS = {
171171
*/
172172
export function sample(logits: Tensor, ctx: SamplingContext, config: SamplingConfig = {}): number {
173173
'worklet';
174+
// `??` rather than spreading the defaults, so a field passed as `undefined`
175+
// falls back to its default instead of reaching the native sampler unset.
176+
const { dry, xtc } = config;
174177
const options = {
175-
...DEFAULT_SAMPLING_OPTIONS,
176178
...config,
177-
dry: config.dry && { ...DEFAULT_DRY_OPTIONS, ...config.dry },
178-
xtc: config.xtc && { ...DEFAULT_XTC_OPTIONS, ...config.xtc },
179+
temperature: config.temperature ?? DEFAULT_SAMPLING_OPTIONS.temperature,
180+
repetitionPenalty: config.repetitionPenalty ?? DEFAULT_SAMPLING_OPTIONS.repetitionPenalty,
181+
penaltyLastN: config.penaltyLastN ?? DEFAULT_SAMPLING_OPTIONS.penaltyLastN,
182+
topK: config.topK ?? DEFAULT_SAMPLING_OPTIONS.topK,
183+
topP: config.topP ?? DEFAULT_SAMPLING_OPTIONS.topP,
184+
minP: config.minP ?? DEFAULT_SAMPLING_OPTIONS.minP,
185+
dry: dry && {
186+
...dry,
187+
base: dry.base ?? DEFAULT_DRY_OPTIONS.base,
188+
allowedLength: dry.allowedLength ?? DEFAULT_DRY_OPTIONS.allowedLength,
189+
penaltyLastN: dry.penaltyLastN ?? DEFAULT_DRY_OPTIONS.penaltyLastN,
190+
sequenceBreakers: dry.sequenceBreakers ?? DEFAULT_DRY_OPTIONS.sequenceBreakers,
191+
},
192+
xtc: xtc && { ...xtc, threshold: xtc.threshold ?? DEFAULT_XTC_OPTIONS.threshold },
179193
};
180194
return rnexecutorchJsi.llm.sample(logits, ctx, options);
181195
}

0 commit comments

Comments
 (0)