Can not get your result

[Imposingapple]

After doing the data preprocessing, training and evaluation for humorous headline generation provided by this repo, i get the following result. The BLEU score is 9.0, much lower than you have written in your paper(13.3).
Is there anything wrong?

[jind11]

Have you initialized the model with pre-trained parameters from MASS?

[Imposingapple]

我在s2s_model.py中看到了调用，确实是已经用下载下来的权重初始化过了。请问还有什么可能的导致效果没有您文章里好的原因？请指教，谢谢！

[Imposingapple]

I download the MASS parameters from your link, and at the beginning of the training, the perplexity is at the magnitude of 10^6. Does it means that the initialization function in s2s_model.py fails to initialize the parameters?
I saw the scripts in s2s_model.py to initialize the model parameters.

[jind11]

yeap, the initial perplexity shouldn't be that high. have you noticed any warning message about model initialization, something like "modules that are not initialized"?

[Imposingapple]

Here's all the terminal outputs after running the script 'CUDA_VISIBLE_DEVICES=0 ./train_mix_CNN_NYT_X.sh --style
humor', haven't seen any wrong signals:

args.distributed_init_method: None
args:
activation_dropout = 0.1
activation_fn = gelu
adam_betas = (0.9, 0.98)
adam_eps = 1e-08
adaptive_input = False
adaptive_input_cutoff = None
adaptive_input_factor = 4
adaptive_softmax_cutoff = None
adaptive_softmax_dropout = 0
adaptive_softmax_factor = 4
arch = transformer_mix_base
attention_dropout = 0.1
best_checkpoint_metric = loss
bpe = None
bucket_cap_mb = 25
char_embedder_highway_layers = 2
character_embedding_dim = 4
character_filters = [(1, 64), (2, 128), (3, 192), (4, 256), (5, 256), (6, 256), (7, 256)]
clip_norm = 0.0
cpu = False
criterion = label_smoothed_cross_entropy
curriculum = 0
dae_styles = humor
data = data/CNN_NYT/processed:data/humor/processed
dataset_impl = None
ddp_backend = no_c10d
decoder_attention_heads = 12
decoder_embed_dim = 768
decoder_ffn_embed_dim = 3072
decoder_langtok = False
decoder_layerdrop = 0
decoder_layers = 6
decoder_layers_to_keep = None
decoder_normalize_before = False
decoder_output_dim = 768
device_id = 0
disable_validation = False
distributed_backend = nccl
distributed_init_method = None
distributed_no_spawn = False
distributed_port = -1
distributed_rank = 0
distributed_world_size = 1
divide_decoder_embed_norm = False
divide_decoder_encoder_attn_norm = False
divide_decoder_encoder_attn_query = True
divide_decoder_final_norm = True
divide_decoder_self_attn_norm = True
divide_decoder_self_attn_query = False
dropout = 0.2
empty_cache_freq = 0
encoder_attention_heads = 12
encoder_embed_dim = 768
encoder_ffn_embed_dim = 3072
encoder_langtok = None
encoder_layers = 6
fast_stat_sync = False
find_unused_parameters = False
fix_batches_to_gpus = False
fixed_validation_seed = None
fp16 = True
fp16_init_scale = 128
fp16_scale_tolerance = 0.0
fp16_scale_window = None
keep_interval_updates = -1
keep_last_epochs = -1
label_smoothing = 0.1
lambda_denoising_config = 0.5
lambda_parallel_config = 0.5
lang_pairs = src-tgt
layernorm_embedding = True
lazy_load = False
left_pad_source = True
left_pad_target = False
load_alignments = False
load_from_pretrained_model = pretrained_model/MASS/mass-base-uncased.pt
log_format = None
log_interval = 1000
lr = [0.0005]
lr_scheduler = inverse_sqrt
max_epoch = 6
max_sentences = None
max_sentences_valid = None
max_source_positions = 512
max_target_positions = 512
max_tokens = 3072
max_tokens_valid = 3072
max_update = 0
max_word_shuffle_distance = 5.0
maximize_best_checkpoint_metric = False
memory_efficient_fp16 = False
min_loss_scale = 0.0001
min_lr = 1e-09
model_lang_pairs = ['src-tgt', 'humor-humor']
no_decoder_final_norm = True
no_epoch_checkpoints = False
no_last_checkpoints = False
no_progress_bar = False
no_save = False
no_save_optimizer_state = False
no_scale_embedding = False
num_workers = 1
optimizer = adam
optimizer_overrides = {}
raw_text = False
required_batch_size_multiple = 8
reset_dataloader = False
reset_lr_scheduler = False
reset_meters = False
reset_optimizer = False
restore_file = checkpoint_last.pt
save_dir = tmp/humor
save_interval = 1
save_interval_updates = 0
seed = 1
sentence_avg = False
share_all_embeddings = True
share_decoder_input_output_embed = True
skip_invalid_size_inputs_valid_test = True
source_lang = None
target_lang = None
task = translation_mix
tensorboard_logdir =
threshold_loss_scale = None
tie_adaptive_proj = False
tie_adaptive_weights = False
tokenizer = None
train_subset = train
truncate_source = False
update_freq = [4]
upsample_primary = 1
use_bmuf = False
user_dir = mass
valid_subset = valid
validate_interval = 1
warmup_init_lr = 1e-07
warmup_updates = 4000
weight_decay = 0.0
word_blanking_prob = 0.2
word_dropout_prob = 0.2
| [src] dictionary: 30522 types
| [tgt] dictionary: 30522 types
| [humor] dictionary: 30522 types
| loaded 3000 examples from: data/CNN_NYT/processed/valid.src-tgt.src
| loaded 3000 examples from: data/CNN_NYT/processed/valid.src-tgt.tgt
| data/CNN_NYT/processed valid src-tgt 3000 examples
| loaded 9651 examples from: data/humor/processed/valid.humor-None.humor
| loaded 9651 examples from: data/humor/processed/valid.humor-None.humor
| denoising-humor: data/humor/processed valid 9651 examples
loading pretrained model from: pretrained_model/MASS/mass-base-uncased.pt
TransformerMixModel(
(encoder): TransformerEncoder(
(embed_tokens): Embedding(30522, 768, padding_idx=0)
(embed_positions): LearnedPositionalEmbedding(513, 768, padding_idx=0)
(layers): ModuleList(
(0): TransformerEncoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(1): TransformerEncoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(2): TransformerEncoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(3): TransformerEncoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(4): TransformerEncoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(5): TransformerEncoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
(emb_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(decoder): TransformerMixDecoder(
(embed_tokens): Embedding(30522, 768, padding_idx=0)
(embed_positions): LearnedPositionalEmbedding(513, 768, padding_idx=0)
(layers): ModuleList(
(0): TransformerMixDecoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(encoder_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): ModuleList(
(0): Linear(in_features=768, out_features=768, bias=True)
(1): Linear(in_features=768, out_features=768, bias=True)
)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(encoder_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
(1): TransformerMixDecoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(encoder_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): ModuleList(
(0): Linear(in_features=768, out_features=768, bias=True)
(1): Linear(in_features=768, out_features=768, bias=True)
)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(encoder_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
(2): TransformerMixDecoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(encoder_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): ModuleList(
(0): Linear(in_features=768, out_features=768, bias=True)
(1): Linear(in_features=768, out_features=768, bias=True)
)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(encoder_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
(3): TransformerMixDecoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(encoder_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): ModuleList(
(0): Linear(in_features=768, out_features=768, bias=True)
(1): Linear(in_features=768, out_features=768, bias=True)
)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(encoder_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
(4): TransformerMixDecoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(encoder_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): ModuleList(
(0): Linear(in_features=768, out_features=768, bias=True)
(1): Linear(in_features=768, out_features=768, bias=True)
)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(encoder_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
(5): TransformerMixDecoderLayer(
(self_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): Linear(in_features=768, out_features=768, bias=True)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(self_attn_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
(encoder_attn): MultiheadAttention(
(k_proj): Linear(in_features=768, out_features=768, bias=True)
(v_proj): Linear(in_features=768, out_features=768, bias=True)
(q_proj): ModuleList(
(0): Linear(in_features=768, out_features=768, bias=True)
(1): Linear(in_features=768, out_features=768, bias=True)
)
(out_proj): Linear(in_features=768, out_features=768, bias=True)
)
(encoder_attn_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(fc1): Linear(in_features=768, out_features=3072, bias=True)
(fc2): Linear(in_features=3072, out_features=768, bias=True)
(final_layer_norm): ModuleList(
(0): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
(1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
)
(emb_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
)
)
| model transformer_mix_base, criterion LabelSmoothedCrossEntropyCriterion
| num. model params: 127031808 (num. trained: 127031808)
| training on 1 GPUs
| max tokens per GPU = 3072 and max sentences per GPU = None
| no existing checkpoint found tmp/humor/checkpoint_last.pt
| loading train data for epoch 0
| loaded 141135 examples from: data/CNN_NYT/processed/train.src-tgt.src
| loaded 141135 examples from: data/CNN_NYT/processed/train.src-tgt.tgt
| data/CNN_NYT/processed train src-tgt 141135 examples
| loaded 480505 examples from: data/humor/processed/train.humor-None.humor
| loaded 480505 examples from: data/humor/processed/train.humor-None.humor
| denoising-humor: data/humor/processed train 480505 examples
| WARNING: 10 samples have invalid sizes and will be skipped, max_positions=OrderedDict([('src-tgt', (512, 512)), ('humor-humor', (512, 512))]), first few sample ids=[141134, 282269, 423404, 480498, 480499, 480500, 480501, 480502, 480503, 480504]
| epoch 001: 0%| | 0/3933 [00:00<?, ?it/s]| WARNING: overflow detected, setting loss scale to: 64.0
| epoch 001: 0%| | 1/3933 [00:01<1:46:55, 1.63s/it]| WARNING: overflow detected, setting loss scale to: 32.0
| epoch 001: 0%| | 2/3933 [00:02<1:06:20, 1.01s/it]| WARNING: overflow detected, setting loss scale to: 16.0
| epoch 001: 0%| | 3/3933 [00:02<56:02, 1.17it/s]| WARNING: overflow detected, setting loss scale to: 8.0
| epoch 001: 0%| | 4/3933 [00:03<51:01, 1.28it/s]

[jind11]

Have you run "evaluate_mix_CNN_NYT_X.sh"? You should use this file for final evaluation.

[Imposingapple]

Yes, of course I did. The first screenshot of this issue is the result after running 'evaluate_mix_CNN_NYT_X.sh' for humor. The hypothesis file to evaluate is already detokenized (sentences with english words, do not have BPE marks).

[Imposingapple]

Dear author, I'm sorry to bother you again. I could not figure out why there's discrepancy between my result and the paper's result yet. I'm sure to run the exact script and use the exact pretrained parameters you provide.
Have you found if there's something wrong with the training or evaluation scripts? Or have you tried running this version of code or found some mistakes of my setting on the above screenshot?
Thank you again for answering!

[zylMozart]

After doing the data preprocessing, training and evaluation for humorous headline generation provided by this repo, i get the following result. The BLEU score is 9.0, much lower than you have written in your paper(13.3). Is there anything wrong?

The BLEU score in my result is 9.79, have you ever solved this discrepancy?