how to export the model

[ghtwht]

How to export model with the format used by lightseq and tortion?

[godweiyang]

You can export the models trained by neurst using the scripts in https://github.com/godweiyang/chatbot/tree/master/export.

The following is the export command:

python3 export.py \
    --model_dir models \
    --output_file model.pb \
    --generation_method topk \
    --topk 4 \
    --length_penalty 0.6 \
    --beam_size 4

[kaierlong]

@godweiyang
Hi, when I use your scripts to export model, there are some errors：

Traceback (most recent call last):
  File "export.py", line 161, in <module>
    app.run(_main, argv=["pseudo.py"])
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "export.py", line 154, in _main
    export_lightseq_model(
  File "export.py", line 127, in export_lightseq_model
    transfer_ckpt2pb(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 256, in transfer_ckpt2pb
    _fill_layer(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 139, in _fill_layer
    assert len(tmp) == 1
AssertionError

the cmd as below

python3 export.py --model_dir ../wmt14_en_de/benchmark_base/ --output_file ../infer/model.pb --beam_size 4 --length_penalty 0.6

and the model is transformer_base.

Any help for this？

[godweiyang]

@godweiyang Hi, when I use your scripts to export model, there are some errors：

Traceback (most recent call last):
  File "export.py", line 161, in <module>
    app.run(_main, argv=["pseudo.py"])
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "export.py", line 154, in _main
    export_lightseq_model(
  File "export.py", line 127, in export_lightseq_model
    transfer_ckpt2pb(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 256, in transfer_ckpt2pb
    _fill_layer(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 139, in _fill_layer
    assert len(tmp) == 1
AssertionError

the cmd as below

python3 export.py --model_dir ../wmt14_en_de/benchmark_base/ --output_file ../infer/model.pb --beam_size 4 --length_penalty 0.6

and the model is transformer_base.

Any help for this？

Can you provide your trainning command and neurst version?

[kaierlong]

@godweiyang
The neurst version is 0.1.0, but I installed it from source use the following command:

git clone -b lightseq https://github.com/bytedance/neurst.git
cd neurst/
pip3 install -e .
pip3 install lightseq-tf2.4.1-cuda11.0.221==2.0.1
pip3 install absl-py
pip3 install tensorflow==2.4.1

• And the training command is:

python3 -m neurst.cli.run_exp --config_paths wmt14_en_de/training_args.yml,wmt14_en_de/translation_bpe.yml,wmt14_en_de/validation_args.yml --hparams_set transformer_base --model_dir wmt14_en_de/benchmark_base

• training_args.yml

entry.class: trainer
entry.params:
  train_steps: 200000
  summary_steps: 200
  save_checkpoint_steps: 1000
  criterion.class: label_smoothed_cross_entropy
  criterion.params:
    label_smoothing: 0.1

• translation_bpe.yml

dataset.class: ParallelTextDataset
dataset.params:
  src_file: wmt14_en_de/train.en.tok.bpe.txt
  trg_file: wmt14_en_de/train.de.tok.bpe.txt
  data_is_processed: True

task.class: translation
task.params:
  batch_by_tokens: True
  batch_size: 4096
  max_src_len: 128
  max_trg_len: 128
  src_data_pipeline.class: TextDataPipeline
  src_data_pipeline.params:
    language: en
    tokenizer: moses
    subtokenizer: bpe
    subtokenizer_codes: wmt14_en_de/codes.bpe
    vocab_path: wmt14_en_de/vocab.en
  trg_data_pipeline.class: TextDataPipeline
  trg_data_pipeline.params:
    language: de
    tokenizer: moses
    subtokenizer: bpe
    subtokenizer_codes: wmt14_en_de/codes.bpe
    vocab_path: wmt14_en_de/vocab.de

• validation_args.yml

validator.class: SeqGenerationValidator
validator.params:
  eval_dataset: ParallelTextDataset
  eval_dataset.params:
    src_file: wmt14_en_de/newstest2013.en.txt
    trg_file: wmt14_en_de/newstest2013.de.txt
  eval_batch_size: 64
  eval_start_at: 5000
  eval_steps: 1000
  eval_criterion: label_smoothed_cross_entropy
  eval_search_method: beam_search
  eval_search_method.params:
    beam_size: 4
    length_penalty: 0.6
    maximum_decode_length: 160
    extra_decode_length: 50
  eval_metric: bleu
  eval_top_checkpoints_to_keep: 10
  eval_auto_average_checkpoints: True

[godweiyang]

@godweiyang The neurst version is 0.1.0, but I installed it from source use the following command:

git clone -b lightseq https://github.com/bytedance/neurst.git
cd neurst/
pip3 install -e .
pip3 install lightseq-tf2.4.1-cuda11.0.221==2.0.1
pip3 install absl-py
pip3 install tensorflow==2.4.1

• And the training command is:

python3 -m neurst.cli.run_exp --config_paths wmt14_en_de/training_args.yml,wmt14_en_de/translation_bpe.yml,wmt14_en_de/validation_args.yml --hparams_set transformer_base --model_dir wmt14_en_de/benchmark_base

• training_args.yml

entry.class: trainer
entry.params:
  train_steps: 200000
  summary_steps: 200
  save_checkpoint_steps: 1000
  criterion.class: label_smoothed_cross_entropy
  criterion.params:
    label_smoothing: 0.1

• translation_bpe.yml

dataset.class: ParallelTextDataset
dataset.params:
  src_file: wmt14_en_de/train.en.tok.bpe.txt
  trg_file: wmt14_en_de/train.de.tok.bpe.txt
  data_is_processed: True

task.class: translation
task.params:
  batch_by_tokens: True
  batch_size: 4096
  max_src_len: 128
  max_trg_len: 128
  src_data_pipeline.class: TextDataPipeline
  src_data_pipeline.params:
    language: en
    tokenizer: moses
    subtokenizer: bpe
    subtokenizer_codes: wmt14_en_de/codes.bpe
    vocab_path: wmt14_en_de/vocab.en
  trg_data_pipeline.class: TextDataPipeline
  trg_data_pipeline.params:
    language: de
    tokenizer: moses
    subtokenizer: bpe
    subtokenizer_codes: wmt14_en_de/codes.bpe
    vocab_path: wmt14_en_de/vocab.de

• validation_args.yml

validator.class: SeqGenerationValidator
validator.params:
  eval_dataset: ParallelTextDataset
  eval_dataset.params:
    src_file: wmt14_en_de/newstest2013.en.txt
    trg_file: wmt14_en_de/newstest2013.de.txt
  eval_batch_size: 64
  eval_start_at: 5000
  eval_steps: 1000
  eval_criterion: label_smoothed_cross_entropy
  eval_search_method: beam_search
  eval_search_method.params:
    beam_size: 4
    length_penalty: 0.6
    maximum_decode_length: 160
    extra_decode_length: 50
  eval_metric: bleu
  eval_top_checkpoints_to_keep: 10
  eval_auto_average_checkpoints: True

Can you undo the comment in https://github.com/godweiyang/chatbot/blob/master/export/transformer_proto_utils.py#L137-L138, and output the two variables?

[kaierlong]

@godweiyang
the new errors as below

2021-12-27 15:27:32.058066: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I1227 15:27:35.874068 139747470812992 flags_core.py:493] ==========================================================================
I1227 15:27:35.874294 139747470812992 flags_core.py:494] Parsed all matched flags:
I1227 15:27:35.874423 139747470812992 flags_core.py:499]  model_dir: ../wmt14_en_de/benchmark_base/     # (default: None) The path to the well-trained checkpoint.
I1227 15:27:35.874494 139747470812992 flags_core.py:499]  output_file: ../infer/model.pb     # (default: None) The path to saving the extracted weights.
I1227 15:27:35.874563 139747470812992 flags_core.py:499]  generation_method: beam_search     # (default: beam_search) The generation method: beam_search (default), topk or topp.
I1227 15:27:35.874632 139747470812992 flags_core.py:499]  beam_size: 4     # (default: None) The beam width of sequence generation for Transformer model.
I1227 15:27:35.874716 139747470812992 flags_core.py:499]  length_penalty: 0.6     # (default: -1) The length penalty of sequence generation for Transformer model.
I1227 15:27:35.874784 139747470812992 flags_core.py:499]  max_seq_len: 256     # (default: 256) The maximum sequence length.
I1227 15:27:35.874855 139747470812992 flags_core.py:499]  extra_decode_length: 50     # (default: 50) The extra decoding length of sequence generation for Transformer model.
I1227 15:27:35.874926 139747470812992 flags_core.py:499]  topk: 4     # (default: 4) topk for sampling, only 1,2,4,...,32 are valid
I1227 15:27:35.874995 139747470812992 flags_core.py:499]  topp: 0.75     # (default: 0.75) topp for sampling
I1227 15:27:35.875064 139747470812992 flags_core.py:499]  use_aligned_penalty: False     # (default: False) Whether to use aligned length penalty
I1227 15:27:35.875136 139747470812992 flags_core.py:499]  align_len_ratio: 1.0     # (default: 1.0) The length ratio against source sequence
I1227 15:27:35.875207 139747470812992 flags_core.py:499]  align_len_bias: 3.0     # (default: 3.0) The length bias
I1227 15:27:35.875275 139747470812992 flags_core.py:499]  align_len_alpha: 1.0     # (default: 1.0) The exponent of length penalty
I1227 15:27:35.875342 139747470812992 flags_core.py:524]
I1227 15:27:35.875405 139747470812992 flags_core.py:525] Other flags:
I1227 15:27:35.875470 139747470812992 flags_core.py:527]  config_paths: None
I1227 15:27:35.875535 139747470812992 flags_core.py:542] ==========================================================================
I1227 15:27:35.875828 139747470812992 export.py:116] Extracting weights from ../wmt14_en_de/benchmark_base/ to ../infer/model.pb
I1227 15:27:35.891601 139747470812992 configurable.py:312] Loading models configs from ../wmt14_en_de/benchmark_base/model_configs.yml
I1227 15:27:35.891697 139747470812992 export.py:120] Extract and export Transformer weights using method: beam_search.
I1227 15:27:35.891793 139747470812992 transformer_proto_utils.py:234] Using /mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer.proto
self_attention_prepost_wrapper gamma
[]
Traceback (most recent call last):
  File "export.py", line 161, in <module>
    app.run(_main, argv=["pseudo.py"])
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "export.py", line 154, in _main
    export_lightseq_model(
  File "export.py", line 127, in export_lightseq_model
    transfer_ckpt2pb(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 256, in transfer_ckpt2pb
    _fill_layer(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 139, in _fill_layer
    assert len(tmp) == 1
AssertionError

[godweiyang]

@godweiyang the new errors as below

2021-12-27 15:27:32.058066: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
I1227 15:27:35.874068 139747470812992 flags_core.py:493] ==========================================================================
I1227 15:27:35.874294 139747470812992 flags_core.py:494] Parsed all matched flags:
I1227 15:27:35.874423 139747470812992 flags_core.py:499]  model_dir: ../wmt14_en_de/benchmark_base/     # (default: None) The path to the well-trained checkpoint.
I1227 15:27:35.874494 139747470812992 flags_core.py:499]  output_file: ../infer/model.pb     # (default: None) The path to saving the extracted weights.
I1227 15:27:35.874563 139747470812992 flags_core.py:499]  generation_method: beam_search     # (default: beam_search) The generation method: beam_search (default), topk or topp.
I1227 15:27:35.874632 139747470812992 flags_core.py:499]  beam_size: 4     # (default: None) The beam width of sequence generation for Transformer model.
I1227 15:27:35.874716 139747470812992 flags_core.py:499]  length_penalty: 0.6     # (default: -1) The length penalty of sequence generation for Transformer model.
I1227 15:27:35.874784 139747470812992 flags_core.py:499]  max_seq_len: 256     # (default: 256) The maximum sequence length.
I1227 15:27:35.874855 139747470812992 flags_core.py:499]  extra_decode_length: 50     # (default: 50) The extra decoding length of sequence generation for Transformer model.
I1227 15:27:35.874926 139747470812992 flags_core.py:499]  topk: 4     # (default: 4) topk for sampling, only 1,2,4,...,32 are valid
I1227 15:27:35.874995 139747470812992 flags_core.py:499]  topp: 0.75     # (default: 0.75) topp for sampling
I1227 15:27:35.875064 139747470812992 flags_core.py:499]  use_aligned_penalty: False     # (default: False) Whether to use aligned length penalty
I1227 15:27:35.875136 139747470812992 flags_core.py:499]  align_len_ratio: 1.0     # (default: 1.0) The length ratio against source sequence
I1227 15:27:35.875207 139747470812992 flags_core.py:499]  align_len_bias: 3.0     # (default: 3.0) The length bias
I1227 15:27:35.875275 139747470812992 flags_core.py:499]  align_len_alpha: 1.0     # (default: 1.0) The exponent of length penalty
I1227 15:27:35.875342 139747470812992 flags_core.py:524]
I1227 15:27:35.875405 139747470812992 flags_core.py:525] Other flags:
I1227 15:27:35.875470 139747470812992 flags_core.py:527]  config_paths: None
I1227 15:27:35.875535 139747470812992 flags_core.py:542] ==========================================================================
I1227 15:27:35.875828 139747470812992 export.py:116] Extracting weights from ../wmt14_en_de/benchmark_base/ to ../infer/model.pb
I1227 15:27:35.891601 139747470812992 configurable.py:312] Loading models configs from ../wmt14_en_de/benchmark_base/model_configs.yml
I1227 15:27:35.891697 139747470812992 export.py:120] Extract and export Transformer weights using method: beam_search.
I1227 15:27:35.891793 139747470812992 transformer_proto_utils.py:234] Using /mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer.proto
self_attention_prepost_wrapper gamma
[]
Traceback (most recent call last):
  File "export.py", line 161, in <module>
    app.run(_main, argv=["pseudo.py"])
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 312, in run
    _run_main(main, args)
  File "/mnt/data_0301_12t/xingchaolong/home/pyenvs/env_neurst_lightseq/lib/python3.8/site-packages/absl/app.py", line 258, in _run_main
    sys.exit(main(argv))
  File "export.py", line 154, in _main
    export_lightseq_model(
  File "export.py", line 127, in export_lightseq_model
    transfer_ckpt2pb(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 256, in transfer_ckpt2pb
    _fill_layer(
  File "/mnt/data_0301_12t/xingchaolong/home/codes/trans/neurst/demo/export/transformer_proto_utils.py", line 139, in _fill_layer
    assert len(tmp) == 1
AssertionError

I know. DO NOT use lightseq branch of neurst, otherwise the varibles can not be exported. So use master branch to train the models.

[kaierlong]

@godweiyang OK, I will test later.