[be]More robust upload for large files using multipart form data; [feat] better management of submission files; [be]some be abstrations; [feat] progress bar for upload

[mazhiyi]

• [BE] Use multipart form data upload to resolve upload errors (a formal requests way to fix the issue in use curl for uploading models #79), and also works better for large file submissions
  • Specifically, tested for both 3GB and 6GB submissions on my local machine and didn't have a problem
• [feat] Better management of submission files as requested in Submission Limit Exceeded Warning: Provide Additional Details #83:
  • Submissions are now saved to .dynalab_submissions and excluded from future submissions
  • Submission files are named as "date-filename.tar.gz` in the submissions folder
• [BE] BE abstractions: avoid explicit use of ".dynalab" everywhere and instead getting it from config_handler.dynalab_dir
• [feat] Also implemented the progress bar. Although since after uploading, there's also a transfer to S3 which we cannot access from locally, which means if you upload a big file, the progress bar will be stuck at 100% for a while when it's being further uploaded to S3)

  % dynalab-cli upload -n zm-test-large-mm100-175m-gpu              !10287
Config file validated
Tarballing the project directory...
Uploading files to S3. For large submissions, the progress bar may hang a while even after uploading reaches 100%. Please do not kill it...
upload: 100%|█████████████████████████| 1.50G/1.50G [00:30<00:00, 52.8MB/s]
Failed to submit model zm-test-large-mm100-175m-gpu due to submission limit exceeded. No more than 3 submissions allowed every 24 hours for task flores_small1.
You can inspect the prepared model submission locally at .dynalab_submissions/Jun-28-2021-15-36-43-zm-test-large-mm100-175m-gpu.tar.gz

Also uploaded a ~6G file to devserver, no problem

  % dynalab-cli upload -n zm-test-large-mm100-175m-gpu              !10364
Config file validated
Tarballing the project directory...
Uploading files to S3. For large submissions, the progress bar may hang a while even after uploading reaches 100%. Please do not kill it...
Upload: 100%|█████████████████████████| 5.98G/5.98G [31:52<00:00, 3.35MB/s]
Your model zm-test-large-mm100-175m-gpu has been uploaded to S3 and will be deployed shortly. You will get an email notification when your model is available on Dynabench.
You can inspect the prepared model submission locally at .dynalab_submissions/Jun-28-2021-18-47-56-zm-test-large-mm100-175m-gpu.tar.gz

[mazhiyi]

cc @gwenzek

[TristanThrush]

LGTM