Hi,
I am trainin on 4 H100 GPUs and encountering an issue were the model outputs gibberish.
The model produced a valid output then a repetitive chatacter loop, and the pattern repeats
--- DEBUG: RAW MODEL OUTPUT START ---
C
--- DEBUG: RAW MODEL OUTPUT START ---
< to for ( with, and, (2 ( for C, and, ( B ( A C, or (, at. ( p. , or. and ( and D ( , , and, for (4, C in A L., de (4. in, and in A '3 (.5 2.8 and and A ( ( ( in, and,, as B. ( ( (3. (. ( of (, ( C. ( C, and, in,, to ( de ( C ( at (4 in for. 2 C,, for ( (1 for B to to as, ( ( ( ( ( ( ( in ( ( at by (11 A [05 and p and in26, and ( (, (. ( a, ( and C A ( ( ( to to,2 ( ( 3 and on while ( [2. at, ( ( and for ( gunshot ( (, A, by - ( -, (4 (, [ (, as in, ( ( or A, to ( A and B as ( as (0 in ', for (, (1 I in cl - (1 ( (2 as and, I ( with., B ( '. a " ( (5 (,, and ( (, as (4 in (. P. as in ( ( (1. P to. P of and to, and ( ( for ( (, if ( (,. is A. ( (26 of p, if, ( and. is at (:, a^2, ( or I, of. ( ( and (: and to (5: ( (: at ( ( [0 ( (1 (, ( for (, (2 for, in - rec, for ( A, and if at ( for ( ( (. ( in. and gunshot, with, (,. or ( to by, ' (: (, B ( I ( I (. ( ( at:21 to be ( as as, ( (. or for ( ( of in or at, to (1.3 ( ( (,, ( A ',5 2,, if ( '2., 1 and to.: , as, in ( ( ( in and , is. (, A. ( in ( ( ( B and of. and for ( I and, and, C (, for in ( ( ( and by, or while (, and,, in, 27 (5. in ( (, ( of in as and for de ( if as,, 4 A ( for ( at, as de: and ( in, and the, I p, as is A, (, ( ( ( as p and (. A and (5 ( (4 in at,., as, of B to (3 ( ( (, (,1 at. ( at if. (, I ( ( and,, and, in (. for ( and ( (, (1 (, ( ( (,. in, D ( [., by ( B 5, 11 A is (. at a valid C, (1., 2 at at or. (以下 ( and for (: B in for the (1 (: ( ( ( and in, and in,, , C. to - to at at, for, in to,, to ( ( (, ( de,, in at a specific with ( (, B. (, ( and, ( at ( (8: ( A of,, - (: B. ( ( ( ( (8,, at ( (. and,, ( of as it ( - ( (1 . C,,. 1 A is and (1 in to 8,, in. 3: I and ( ( (, to ( ( L and, at, ( ( ( ( and..3 for ( ' P. in, 2: ( ( in for (,,.1. is (, to ( [ at by F, B, ( I cl, with, C, ( 1 , - (2., p to.. ( ( and ( with [ (1. of D
I was not able to unistall Flash Attention 2 and used SDPA instead with BF16.
Does the model require attn_implementation="flash_attention_2" to maintain numerical stability?
Should I be applying a specific repetition_penalty or temperature setting in the generation_config to prevent this degeneration?
Thank you for any help provided.
Hi,
I am trainin on 4 H100 GPUs and encountering an issue were the model outputs gibberish.
The model produced a valid output then a repetitive chatacter loop, and the pattern repeats
--- DEBUG: RAW MODEL OUTPUT START ---
C
--- DEBUG: RAW MODEL OUTPUT START ---
< to for ( with, and, (2 ( for C, and, ( B ( A C, or (, at. ( p. , or. and ( and D ( , , and, for (4, C in A L., de (4. in, and in A '3 (.5 2.8 and and A ( ( ( in, and,, as B. ( ( (3. (. ( of (, ( C. ( C, and, in,, to ( de ( C ( at (4 in for. 2 C,, for ( (1 for B to to as, ( ( ( ( ( ( ( in ( ( at by (11 A [05 and p and in26, and ( (, (. ( a, ( and C A ( ( ( to to,2 ( ( 3 and on while ( [2. at, ( ( and for ( gunshot ( (, A, by - ( -, (4 (, [ (, as in, ( ( or A, to ( A and B as ( as (0 in ', for (, (1 I in cl - (1 ( (2 as and, I ( with., B ( '. a " ( (5 (,, and ( (, as (4 in (. P. as in ( ( (1. P to. P of and to, and ( ( for ( (, if ( (,. is A. ( (26 of p, if, ( and. is at (:, a^2, ( or I, of. ( ( and (: and to (5: ( (: at ( ( [0 ( (1 (, ( for (, (2 for, in - rec, for ( A, and if at ( for ( ( (. ( in. and gunshot, with, (,. or ( to by, ' (: (, B ( I ( I (. ( ( at:21 to be ( as as, ( (. or for ( ( of in or at, to (1.3 ( ( (,, ( A ',5 2,, if ( '2., 1 and to.: , as, in ( ( ( in and , is. (, A. ( in ( ( ( B and of. and for ( I and, and, C (, for in ( ( ( and by, or while (, and,, in, 27 (5. in ( (, ( of in as and for de ( if as,, 4 A ( for ( at, as de: and ( in, and the, I p, as is A, (, ( ( ( as p and (. A and (5 ( (4 in at,., as, of B to (3 ( ( (, (,1 at. ( at if. (, I ( ( and,, and, in (. for ( and ( (, (1 (, ( ( (,. in, D ( [., by ( B 5, 11 A is (. at a valid C, (1., 2 at at or. (以下 ( and for (: B in for the (1 (: ( ( ( and in, and in,, , C. to - to at at, for, in to,, to ( ( (, ( de,, in at a specific with ( (, B. (, ( and, ( at ( (8: ( A of,, - (: B. ( ( ( ( (8,, at ( (. and,, ( of as it ( - ( (1 . C,,. 1 A is and (1 in to 8,, in. 3: I and ( ( (, to ( ( L and, at, ( ( ( ( and..3 for ( ' P. in, 2: ( ( in for (,,.1. is (, to ( [ at by F, B, ( I cl, with, C, ( 1 , - (2., p to.. ( ( and ( with [ (1. of D
I was not able to unistall Flash Attention 2 and used SDPA instead with BF16.
Does the model require attn_implementation="flash_attention_2" to maintain numerical stability?
Should I be applying a specific repetition_penalty or temperature setting in the generation_config to prevent this degeneration?
Thank you for any help provided.