I have set up bench2drive and am running routes on SimLingo. I noticed that route and speed predictions vary across runs even with identical inputs.
I performed a deep inspection and found that all modules exhibit deterministic behavior up until the final call: features, logits = self.language_model.forward(input_embed_concat)
Suspecting that the LoRA (PEFT) dropout was introducing stochasticity, I replaced the dropout layers with nn.Identity. However, the output of the language model remains non-deterministic during the final pass.
Aside from dropout, are there other potential sources of randomness in the LLM forward pass I should investigate to enforce strict determinism?
I have set up bench2drive and am running routes on SimLingo. I noticed that route and speed predictions vary across runs even with identical inputs.
I performed a deep inspection and found that all modules exhibit deterministic behavior up until the final call: features, logits = self.language_model.forward(input_embed_concat)
Suspecting that the LoRA (PEFT) dropout was introducing stochasticity, I replaced the dropout layers with nn.Identity. However, the output of the language model remains non-deterministic during the final pass.
Aside from dropout, are there other potential sources of randomness in the LLM forward pass I should investigate to enforce strict determinism?