RNN Transducer Loss

[vincentqb]

This issue is to track the follow-up work to #1137, which introduced rnnt_loss and RNNTLoss as a prototype in torchaudio.prototype.transducer using HawkAaron's warp-transducer.

• Update documentation
  • Guard documentation (e.g. with conditional doc build)
  • Write e.g. torchtext (Add documentation for RNN transducer #1171)
• Extend guards for prototype
  • Guard prototype python files by omitting them from torchaudio, see also Add RNN Transducer Loss for CPU #1137 (comment)
  • Guard building third party transducer even if not added as an extension (Clean up transducer build #1159)
  • Enable building transducer in nightlies only, disable for release.
• Update building process
  • Pass along the DEBUG flag to cmake
  • Remove hardcoded O2/O3 optimization, see Add RNN Transducer Loss for CPU #1137 (comment) (Clean up transducer build #1159)
  • Build within same folders as libsox, Add RNN Transducer Loss for CPU #1137 (comment) and Add RNN Transducer Loss for CPU #1137 (comment) (Clean up transducer build #1159)
  • Move libsox to a third_party subfolder as suggested in Add RNN Transducer Loss for CPU #1137 (comment) (Clean up libsox build and directory location #1161).
  • Add GPU implementation and compilation. (see Add GPU RNNT Loss #1483)
  • Add USE_CUDA option for user: build currently depends on presence of device, see here, and pytorch.
  • Add CUDA build binaries, Add CUDA binary builds #1497
• Modernization
  • Migrate the checks to C++.
  • Add autograd test RNN Transducer Loss Autograd Test #1532
  • Add Torchscriptability test (attempt, internal).
  • Investigate using AT_DISPATCH_FLOATING_TYPES.
  • Update bindings to remove pytorch deprecation warnings. (Update data pointer in RNN-T loss #1160)
  • Refactor and update the API, see warprnnt and internal.
  • Add support for float16.
  • rnnt loss should not capture the gradient here. (Should rnnt loss custom C++ autograd function return the gradient?)
  • Remove numpy test utilities from tests
  • Replace change of parameter to assertion here

cc @astaff, internal

[maxwellzh]

Is there a plan to support the packed layout logits of RNNT loss?

Ref: Sec 3.1 https://arxiv.org/abs/1909.12415